ChefEssentials

TableofContents

ChefEssentials

Credits

AbouttheAuthor

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffersandmore

WhySubscribe?

FreeAccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Errata

Piracy

Questions

1.InstallingChef

Terminology

WorkingwithChef

Installingchef-solo

TheRubygem

Managinggems

Verifyingthatchef-soloworks

InstallingaChefserver

Requirementsandrecentchanges

Installationrequirements

Whatyouwillbeinstalling

Gettingtheinstaller

Installationoutline

InstallingonUbuntu

Downloadingthepackage

Installingthepackage

InstallingonRedHatEnterpriseLinux

Downloadingthepackage

ConfiguringaChefserver

Understandinghowchef-server-ctlworks

What’shappeningonmyserver?

Verifyingthattheservicesarerunning

Validatingthatyourserviceisworking

Ensuringthatyourknifeconfigurationworks

Summary

2.ModelingYourInfrastructure

GettingtoknowChef

Modelingyourinfrastructure

Roles

Definingroles

Awebapplicationservicerole

Animage-processingrole

Animagesearchrole

APostgreSQLservicerole

ASolrservicerole

AnOpenSSHservicerole

Implementingarole

Determiningwhichrecipesyouneed

Installingacookbook

Applyingrecipestoroles

Mappingyourrolestonodes

Converginganode

Environments

Organizingyourconfigurationdata

Exampleattributedata

Databags

Knowingwhentousedatabags

Large-scaleinfrastructure

Summary

3.IntegratingwiththeCloud

Leveragingthecloud

AmazonEC2

InstallingtheEC2knifeplugin

SettingupEC2authentication

Provisioninganinstance

Bootstrappingtheinstance

Terminatingtheinstance

RemovingtheChefnode

RackspaceCloud

Provisioninganinstance

Terminatinganinstance

RemovingtheChefnode

Summary

4.WorkingwithCookbooks

Attributes

Multipleattributefiles

Supportingmultipleplatforms

Loadingexternalattributes

Usingattributes

Metadata

Recipes

Resources

Usingresources

Overridingadefaultbehavior

Templates

Whyusetemplates?

AquickERBprimer

ExecutingRuby

Variablereplacement

Thetemplateresource

Thetemplatevariables

Passingvariablestoatemplate

Accessingcomputedconfigurations

Searchingfortemplates

Definitions

Recipes

Developingrecipes

Writingrecipes

Startingoutsmall

Installingasimpleservice

Gettingmoreadvanced

Summary

5.TestingYourRecipes

Testingrecipes

RSpec

RSpecandChefSpec

Testingbasics

ComparingRSpecwithothertestinglibraries

UsingChefSpec

GettingstartedwithChefSpec

InstallingChefSpec

LockingyourdependenciesinRuby

CreatingasimplerecipeandamatchingChefSpectest

WritingaChefSpectest

Buildingyourrecipe

Executingtests

Understandingfailures

Expandingyourtests

Multipleexamplesinaspectest

Testingformultipleplatforms

Summary

6.FromDevelopmenttoDeployment

Describingthesetup

DeployingsoftwarewithChef

Configuringyourlocalenvironment

ModelingasimplePythonapplication

Managingthecookbooks

Downloadingcookbooks

Lookingatthedatabaserecipe

Lookingatyourapplicationdeploymentcookbook

Preparingthedirectories

ConstructingyourPythonvirtualenvironment

Checkingthesourcecode

Installinganyextradependencies

ManagingdependenciesinChef

Managingdependencieselsewhere

UsingPython’srequirementsfile

Configuringyourapplication

Keepingyourapplicationrunning

Definingroles

Creatingthebaseserverrole

Creatingthedatabaseserverrole

Creatingthewebserverrole

Addingusers

ProvisioningEC2instances

Configuringthedatabasehost

Configuringthewebserver

Deployingyoursoftware

Manuallydeployingupdates

Automatingdeployment

Summary

7.BeyondBasicRecipesandCookbooks

Managingusers

Evolutionofashelluserrecipe

Storingdataindatabags

Creatingadatabagforusers

Searchingfordata

Searchinginsiderecipes

Enhancingyourusercookbook

DistributingSSHkeys

Templatingtheauthorizedkeys

Addingdeploymentkeys

Writingcustomextensions

Developingacustomdefinition

Organizingyourcode

WritingadefinitionforusingPIP

Definingafullapplicationtemplate

Buildingaresource

Definingtheresource

Implementingtheprovider

Modifyingresources

Loadinganexistingresource

Declaringthataresourcewasupdated

Workingwithdatabags

Securingyourdatabags

Secretkeys

Encryptingyourdata

Decryptingyourdata

Storingkeysonnodes

Searchingyourdata

Searchingyourdatabagswithknife

Searchingyourdatabagsfromarecipe

Queryingyourdata

Managingmultiplemachineswithsearchqueries

Summary

8.ExtrasYouNeedtoKnow

VagrantandChef-solo

InstallingVagrant

ProvisioninganewhostwithVagrant

BootingyourVagrantimage

CombiningVagrantwithChef-solo

UnderstandingthelimitationsofChef-solo

ConfiguringChef-solo

TellingChef-solowhattorun

UsingrolesanddatabagswithChef-solo

InjectingcustomJSONdata

Providingacustomnodename

GettingtoknowtheChefshell

UsingtheChefshell

Thestandalonemode

Thesolomode

Theclientmode

InteractingwiththeChefserverusingtheshell

Interactingwithdata

Searchingyourdata

Editingyourdata

Transformingdata

ExecutingrecipeswithChefshell

Creatingarecipeintheshell

Definingnodeattributes

Settingattributes

Accessingattributes

Usingconfigurationblocks

Interactivelyexecutingrecipes

DebuggingwiththeChefshell

Usingthebreakpointresource

Integrationtesting

UsingTestKitchen

InstallingTestKitchen

TestingwithTestKitchen

Buildingasimplecookbook

Preparingyourcookbookforthekitchen

Testingyournewcookbook

Provisioningtheinstance

Convergingthenewlycreatedinstance

Writingasimpletest

Combiningallthesteps

ExtendingChef

WritinganOhaiplugin

AnoteaboutwritingOhaiplugins

ChefwithCapistrano

Automationandintegration

Automatedupdatesanddeployments

Summary

Index

ChefEssentials

ChefEssentialsCopyright©2014PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:September2014

Productionreference:1190914

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78398-304-9

www.packtpub.com

CoverimagebyPrashantTimappaShetty(<sparkling.spectrum.123@gmail.com>)

CreditsAuthor

JohnEwart

Reviewers

JoshuaBlack

LaurenMalhoit

EricMaxey

CommissioningEditor

EdwardGordon

AcquisitionEditor

LlewellynRozario

ContentDevelopmentEditor

GovindanK

TechnicalEditor

ShubhangiDhamgaye

CopyEditors

RoshniBanerjee

MradulaHegde

GladsonMonteiro

ProjectCoordinator

SageerParkar

Proofreaders

SimranBhogal

MariaGould

AmeeshaGreen

LindaMorris

Indexer

RekhaNair

ProductionCoordinator

ArvindkumarGupta

CoverWork

ArvindkumarGupta

AbouttheAuthorJohnEwartisasystemarchitect,softwaredeveloper,andlecturer.Hehasdesignedandtaughtcoursesatavarietyofinstitutions,includingtheUniversityofCalifornia,TheCaliforniaStateUniversity,andlocalcommunitycolleges.Thesecoursescoverawiderangeofcomputersciencetopics,includingJava,datastructuresandalgorithms,operatingsystemsfundamentals,UnixandLinuxsystemadministration,andwebapplicationdevelopment.Inadditiontoworkingandteaching,hemaintainsandcontributestoanumberofopensourceprojects.HecurrentlyresidesinRedmond,Washington,withhiswife,Mary,andtheirtwochildren.

AbouttheReviewersJoshuaBlackhasbeenworkingwithcomputersprofessionallyfor20years.Hehasawiderangeofexperienceandexpertise,whichincludessystemsandnetworkadministration,mobileappdevelopment,andproductionwebapplications.HeearnedaBSdegreeinComputerSciencewithaminorinMathfromCaliforniaStateUniversity,Chico,in2005.HecurrentlyresidesinChico,California,withhiswife,Rachel,andtheirfourchildren.

LaurenMalhoithasbeeninthefieldofITforover10yearsandhasacquiredseveraldatacentercertifications.She’scurrentlyatechnicalvirtualizationarchitect,specializinginvirtualizationandstorageindatacenter.ShehasbeenwritingforafewyearsforTechRepublic,TechRepublicPro,andVirtualizationSoftware.AsaCiscoChampion,EMCElect,VMwarevExpert,andPernixPro,shestaysinvolvedinthecommunity.Shealsohostsabi-weeklytechnologypodcastcalledAdaptingIT(http://www.adaptingit.com/).ShehasbeenadelegateforTechFieldDayseveraltimesaswell.Sherecentlypublishedherfirstbook,VMwarevCenterOperationsManagerEssentials,PacktPublishing.

EricMaxeyhasavariedbackgroundinwritingsoftware,includingmakingconsolevideogamesandanalyzingtheadrevenuedata.Heranasmallbusinessthatpioneeredanewkindofbitcoinminingpool,whichhasnowbecomesomethingofastandard.Whennotjackedintothemetaverse,helikestoworkonelectricbicyclesandridethemoffroad.

www.PacktPub.com

Supportfiles,eBooks,discountoffersandmoreYoumightwanttovisitwww.PacktPub.comforsupportfilesanddownloadsrelatedtoyourbook.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

http://PacktLib.PacktPub.com

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcanaccess,readandsearchacrossPackt’sentirelibraryofbooks.

WhySubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,printandbookmarkcontentOndemandandaccessibleviawebbrowser

FreeAccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandviewnineentirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

PrefaceChefisanindispensabletooltomanageyourinfrastructure.Itconsistsofasetoftoolsthataredesignedtoworktogethertoenableyoutomodelandmanageyoursystems.Thisisalargespacetofill,andChefprovidesyouwiththetoolstodothisinaveryflexibleandpowerfulway.Itachievesthisthroughacombinationofservices,endhostagents,awebinterface,andcommand-linetoolsthatworkinunisontodeliveranincrediblesuiteoftools.

Chef’sservicesareresponsibleforstoring,managing,anddistributingdataaboutyourinfrastructurethroughanAPI.Endhostsoftwareagentsthatrunonnodes(managedsystems)areresponsibleforperformingupdatestosystems,andthewebinterface,alongwithcommand-linetools,allowsanadministratortoeditandconsumeinformationthatisvendedbytheAPIservice.

OneofthemostattractivefeaturesofChefisthatyoucanleverageitsAPItoeasilyintegrateexistingtools,oryoucandevelopnewtoolstomeetspecificneeds.AnyorganizationwithamoderatenumberofdeveloperresourcescanharnessthepowerofCheftomanagetheirsystems.Forexample,onecaneasilybuildsoftwaretoimportdatafromChefintoareportingtoolofsomeformanddynamicallyreconfigureinfrastructurebasedonathird-partytool’soutput—thesky’sthelimit.ThisisincrediblyvaluabletoanybodywhohasanexistinginfrastructurebecauseitprovidesaconvenientpathtointegrateChefintotheirenvironment.

ThereareanumberofwaystoaccessChef.Thequickestwayforasingleusertomanagehis/herinfrastructure(virtualmachines,ahandfulofhosts,andsoon)istouseChef-solo,aproductgearedtowardssingle-userenvironments.Inasmallenvironment,settingupahostedserverisagoodwaytomanageinfrastructureautomationamongteammembers.Ifyouneedto,youcanconfigurethehostedenvironmentasahighlyavailablesystemusingloadbalancersandothertechnologies.Alternatively,ifhostingtheserviceyourselfisnotanoption,youcanusehostedChef,asoftware-as-a-service(SaaS)model,thuspayingforaccesstoahostedservice.

Configurationmanagementsoftwarewascreatedtofillaneed—managinginfrastructureisachallengingtask.Regardlessofthescaleyouoperateon,keepingtrackofsoftwareversions,upgradingsystems,andgeneratingconsistentconfigurationdataisalotofwork.Itistemptingtoupdateaconfigurationfileononesystem,onlytoforgettocommitthosechangessomewhere,ortoapplythemtoexistingorfuturehosts.Thisisveryconvenient,butitquicklyleadstoinconsistencybetweenhosts.Whenyouareworkingwithonlyoneortwohosts,thismaybeacceptable.Assuch,asystemgrowsfromafewserverstodozens,hundreds,orpossiblythousands—thistypeofsystemmanagementdoesnotscaleduetotimerequirementsandconfigurationerrorsthatresultfromsizeandcomplexity.

Considerascenariowhereyouaremigratingadatabaseservertoanewhost.Thiswouldinvolve:bringingupanewhost,installingalloftherequiredsoftwareonyourlisttoensureithasparitywiththeoldserver(youdidkeepalist,right?),ensuringthatyourdatabaseserverwasconfiguredwiththesameoptions,puttingthecorrectfirewallrulesin

place,tuningthefilesystem,settingupmonitoringtools,updatingDNSrecordsorchangingwebapplicationconfigurationstopointtothenewhost,andsoon.Nowimaginethat,insteadofoneserverinonedatacenter,youhave10databaseserversin10datacenters,eachwiththeirownIPranges,hardwareconfigurations,andnetworkingrules.Situationssuchasthisareexactlywhysystemconfigurationmanagementsoftwarepackagesweredeveloped:tomakethelivesofsystemadministratorsandengineersmucheasier.

Thisscenario,andmanyotherslikeit,iswhereChefisindispensable.Havingtheabilitytodescribeyourhosts,configurationdata,androles,andthenapplythatacrossasmanyhostsasyoulikemeansthatyoucanmanagelargefleetsofhostsjustaseasilyasyoucanmanageoneortwo.

WhatthisbookcoversChapter1,InstallingChef,introducesyoutothearchitectureofChef,variousinstallationmethods,andaguidetosettingupChef(soloandself-hosted).ItincludesinformationonusinghostedChef(andwhatthatmeansforyourteam)andVagrantwithChef-solo.

Chapter2,ModelingYourInfrastructure,introduceshowtomodelyourinfrastructurewithChefusingyournewlyinstalledsystem.Thischapterwillcovermodelingenvironments,smallandlarge,aswellashowtointegratewithcloudtechnologiesusingChef(AWS,RackspaceCloud,andsoon).

Chapter3,IntegratingwiththeCloud,covershowChefhelpsyouscaleyourinfrastructureusinganycombinationofphysical,virtual,andcloud-hostedsystems.ThischapterdiscusseshowtouseCheftoprovisionandmanagehostsusingcloudprovidersaseasilyasyourlocalsystemsincludingAWSandRackspaceCloud.

Chapter4,WorkingwithCookbooks,covershoweveryChefneedscookbooks—onceyoursystemsarepartofyourChef-managedfleet,youcanbegincollecting,developing,andapplyingrecipestoyourhosts.Itincludesin-depthexplanationsofthestructureanddevelopmentofcookbooksandrecipes,aswellashowtotest,publish,andsharethem.

Chapter5,TestingYourRecipes,focusesononecompellingreasontouseCheftoconfigureyourinfrastructure,thatis,recipesarewritteninRubycodeandcanbetestedasanyprogramwouldbetested.Here,youwilllearnhowtotestyourrecipesthroughavarietyoftestingmechanisms.

Chapter6,FromDevelopmenttoDeployment,covershowtotakeacustomapplicationfromdevelopmenttoaproductiondeploymentwithChef.Itcontainsacompleteexamplethatincludesprovisioningawebserver,databaseserver,andusersaswellasdeployingcodefromsourcecontrol.

Chapter7,BeyondBasicRecipesandCookbooks,delvesintodevelopingextensionstoChefthroughadvancedconcepts,includingcustomprovidersandresourcetypes,usingtheChefsearchengine,advancedscripting,andmore.

Chapter8,ExtrasYouNeedtoKnow,expandsyourknowledgeofhowtoleverageChefforinfrastructureautomation,complexsystemsintegration,andsecurelystoringanddistributingsensitivedatawithChef.

WhatyouneedforthisbookThisbookassumesthatyouarefamiliarwithatleastoneprogramminglanguage(itdoesnotneedtobeacompiledlanguage,andknowledgeofaninterpretedlanguagewillbesuitable.ChefusesRubyforitsdynamic,scriptablecomponentsandanyexperiencewithRubywillbevaluable.However,havingastrongunderstandingofprogramlogicwillprovideyouwiththebackgroundtobeproductivewithChef.

ForthosewhoarenotexpertswithRuby,therewillbeawidearrayofexamplelistingsthatcanbecopieddirectlyandexecutedaspartofthebook’sofferings.ThiswillenableyoutousetheexampleswithoutanypreviousRubyexperience.However,aworkingknowledgeofRubywillbeneededinordertoexpandonthebook’scodeexamplesorwhilewritingyourownrecipesfromscratch.

YouwillbewalkedthroughthestepsrequiredtoinstallChefonaLinux-basedhost.Inordertobeimmediatelysuccessful,youwillneedadministrativeaccesstoahostthatrunsamodernversionofLinux;Ubuntu13.10iswhatwillbeusedfordemonstrationpurposes.Ifyouareamoreexperiencedreader,thenarecentreleaseofalmostanydistributionwillworkjustaswell(butyoumayberequiredtodoalittlebitofextraworkthatisnotoutlinedinthebook).IfyoudonothaveaccesstoadedicatedLinuxhost,avirtualhost(orhosts),runninginsideofvirtualizationsoftware,suchasVirtualBoxwillwork.

Additionally,youwillneedaccesstotheInternettodownloadsoftwarepackagesthatyoudonotalreadyhave,aswellasaninstallationoftheRubyprogramminglanguageVersion1.9orhigher.

WhothisbookisforThisbooktargetsdevelopersandsystemadministratorswhoneedtomanageinfrastructureandarelookingtoautomatetheirsystemmanagement.Thisincludesinfrastructureranginginsizefromsmall-scaleinstallationswithahandfulofhoststomulticontinentcorporateITsystemswithhundredsoreventhousandsofhosts.Anybodywhosejobinvolvesmaintainingsystemswillbenefitfromtheconceptsbeingcovered.

ConventionsInthisbook,youwillfindanumberofstylesoftextthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestyles,andanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Thesearchmethodhasasimilarformattotheknifecommand.”

Ablockofcodeissetasfollows:

all_users=search(:users,'id:*')

users_s=search(:users,'id:s*')

all_nodes=search(:node,'*')

Anycommand-lineinputoroutputiswrittenasfollows:

$knifedatabagshowcredentialsaws

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,inmenusordialogboxesforexample,appearinthetextlikethis:“Oncethere,atablabeledChefServerwillbepresentatthetopofthepage.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedormayhavedisliked.Readerfeedbackisimportantforustodeveloptitlesthatyoureallygetthemostoutof.

Tosendusgeneralfeedback,simplysendane-mailto<feedback@packtpub.com>,andmentionthebooktitleviathesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideonwww.packtpub.com/authors.

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyouwouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheerratasubmissionformlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedonourwebsite,oraddedtoanylistofexistingerrata,undertheErratasectionofthattitle.Anyexistingerratacanbeviewedbyselectingyourtitlefromhttp://www.packtpub.com/support.

PiracyPiracyofcopyrightmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.Ifyoucomeacrossanyillegalcopiesofourworks,inanyform,ontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthors,andourabilitytobringyouvaluablecontent.

QuestionsYoucancontactusat<questions@packtpub.com>ifyouarehavingaproblemwithanyaspectofthebook,andwewilldoourbesttoaddressit.

Chapter1.InstallingChefBeforeyoucanstartusingChef,youwillneedtoinstallit.HereyouwillfindaguidetoinstallChef,andbecauseChefrequiresRuby,someRubyconceptsaswell.Thischapterdiscussesthefollowing:

KeyterminologyandconceptsrelatedtoChefAnoverviewofChef’sarchitectureWorkingwithRubygemsInstallingchef-solo(alocal-onlyenginetouseChef)Abriefexampleonusingchef-soloInstallingtheChefserveronyourownhostVerifyingyourChefinstallation

TerminologyAswithanyothertechnology,Chefhasitsownterminology.Asyouwillsee,Chef’snomenclatureisamixoftechnologicalterms(nodes,workstations,servers,roles,andsoon)andcookingterms(cookbooks,recipes,andsoon).Therearethreeprimaryactorsthatweareconcernedwithatthispoint:nodes,theChefservice,andworkstations.

Node:Anodeisaclientthatappliesrolesandrecipes,asdescribedbytheadministratorintheChefservice(thatis,aserverinyourenvironmentthatisbeingconfiguredviaChef).Thesearetheconsumersoftheconfiguration,theelementsofyourinfrastructure.TheycanbephysicalorvirtualmachinesandcanrunonLinux,Windows,ortechnicallyanyothersystemthatiscapableofrunningRuby(somesystemsmaynotbesupportedoutoftheboxbyChef).Chefservice:TheChefserviceisamulticomponentsystemthatcombinesseveralservicestoprovideitsfunctionality.TheprimaryfunctionalcomponentsareanAPIservice,full-textsearchingviaSolr,persistentstorageusingPostgreSQL,andRabbitMQforinterservicecommunication.Additionally,thereisawebinterfacethatprovidesagraphicaltooltomanagesystemdata.Clients(nodes)usetheAPIservicetodeterminewhichrolesandrecipestoapply,andknife(acommand-linetool)usestheAPItoallowanadministratortoeditandmanagetheirChefconfiguration.Workstation:Aworkstationisahostthatisusedtoissuecommands.AworkstationcanbeaseparatehostoutsideofyourChefserviceinstallation,amanagednode,ortheserverthattheChefcomponentsarerunningon.Thereareavarietyofcommand-linetoolsthatareprovidedtointeractwiththeservice,whichwillbeinstalledontoyourworkstation(s).Recipe:Arecipeisascriptthatdescribesasetofstepstotaketoachieveaspecificgoal.Asanexample,arecipemightdescribehowtodeployyourcustomsoftware,provisionadatabase,oraddahosttoaloadbalancer.Cookbook:Acookbookisacollectionofrecipesthatareusedtocollectivelydescribehowtoinstall,configure,andmanagevariousaspectsofyourinfrastructure.Forexample,acookbookmightdescribehowtoprovisionMySQL,PostgreSQLorApache,manageusers,installprinters,orperformanyothersystemtasks.

WorkingwithChefForsingleusersetups,chef-soloisaversionofthechef-clientthatallowsyoutouseChefwithoutrequiringaccesstoaserver.Chef-solorunslocallyandrequiresthatacookbookandanyofitsdependenciesbepresentonthenodebeingmanaged.Asaresult,chef-soloprovidesalimitedsubsetofthefullchefservermodeofoperation.Mostofthefeaturesthatchef-soloismissingrevolvearoundsearchandcentralizeddatamanagement,whicharenotcriticalformanagingvirtualmachinesorasmallcollectionofnodes.Theinstallationandmaintenanceissimple,butthefeaturesetissmaller.

InstallingtheChefserverwillgiveyouaccesstothefullsetofCheffunctionality.ThismoderequiresaccesstoaLinux-basedhostthatisnetwork-accessiblebythenodesandworkstationsthatwillinteractwiththesystem.ThankstotherecenteffortfromthefolksatChef(formerlyOpscode),theprocessofinstallingChefhasbeengreatlysimplified.Thebenefitsofthisinstallationmodelarethatyougetcentralizedmanagement,search,userauthentication,andsuch,butatthecostofmanagingyourownservice.

IfyouneedthefeaturesofChefbutdonotwanttomaintainyourownserver,hostedChefisagreatoptionforyou.HostedChef(https://manage.opscode.com/signup)givesyouallthefeaturesofaself-hostedChefbutwithouthavingtoworryaboutupgrades,extrahardware,orsystemavailability.Forasmallinfrastructure(uptofivehosts),hostedChefisfreeandagreatwaytogetstarted.Beyondthis,planshaveamonthlyfee,andthepricewillvaryaccordingtothenumberofhostsyouwanttomanage.

Installingchef-soloChef-soloisdesignedforindividualswhodonotneedahostedinstallationforalarge-scaleinfrastructuremanagement.Typicalusecasesofchef-soloincludedevelopersmanagingvirtualmachines,testinstallations,orsmall-scaleinfrastructuremanagement.Theinstallationofchef-soloisassimpleasinstallingasingleRubygem.

TheRubygemForthosewhoarenotintimatelyfamiliarwithRuby,aRubygemisamechanismtopackage,deliver,andmanageRubycode.Thesepackagesmaybelibrariesthatprovidefunctionalityfordevelopers,ortheymaybecomposedonlyofscriptsandtools.Chef-solois,likemanythingsinlife,somewhereinthemiddle.Thegemcontainsasetoflibrariesthatmakeupthecorefunctionalityaswellasasuiteofscriptsthatareusedbyendusers.BeforeyouinstallChef,youshouldconsiderinstallingRubyVersionManager(RVM),rbenv,chruby,oranotherRubymanagerofyourchoicetokeepyourgemcollectionsisolated.

ManaginggemsAgreattooltomanageyourgemsisRVM.ThesimplestwaytoinstallRVMistousetheinstallationscriptprovidedbythedevelopmentteamontheRVMwebsite(http://rvm.io).Thefollowingcommandwilldownloadthescriptandpipeitthroughbash:

curl-sSLhttps://get.rvm.io|bash-sstable

Onceitisinstalled,youwillinitiallyneedtoincludeRVM’sfunctionalityinyourshell:

source~/.rvm/scripts/rvm

Additionally,youmightneedtoaddthepreviouscommandlinetoyourshell’sstartupscripts(suchas~/.bashrcor~/.zshrc,dependingonwhichshellyouuse).OnceRVMisinstalled,youwillwanttoinstallarecentversionofRuby,forexample,Ruby1.9.3:

rvminstall1.9.3

OnceRuby1.9isinstalled,youwillwanttocreateagemset.AgemsetisRVM’swayofisolatinggemsinsideacontainer,anditwillprovideyouwithaplacetoinstallgemsinsuchawaythattheywillnotconflictwithothergems.Thishasthebenefitofallowingyoutoinstallanythingyouwant,withoutrequiringadministrativeprivilegesandkeepinggemsfromconflictingwitheachother.Agemsetcanbecreatedusingthefollowingcommand:

rvmuse1.9.3@chef--create

Thepreviouscommandwillsimultaneouslycreatethegemsetnamedchef(ifitdoesnotexist)foryourinstallationofRuby1.9.3andthensetitastheactivegemset.Onceyoustartusingthisnewgemset,youwillwanttoinstalltheChefgem—thiscontainschef-soloandallthecommand-linetoolsyouneedtoworkwithChef—usingthegemcommand-linetool:

geminstallchef

Verifyingthatchef-soloworksNowthattheChefgemisinstalled,itistimetoverifythateverythingisworkingfine.Inordertousechef-solo,youneedtogivethefollowinginformationtoit:

Whatrecipestoapplybyprovidingarunlistinafilenamednode.jsonWhattherecipesare—thesearestoredincookbooksthatarefoundinthecookbooksdirectoryHowtofindthecookbooksandtherunlistviaafilenamedsolo.rb

Forsimplicity,wewillstoreallofthesefilesinsideofthechefdirectoryinyourhomedirectory.YouarefreetoputthingswhereyouseefitasyoubecomemorecomfortableworkingwithChef.

Inordertoexerciseournewtool,wewilldosomethingsimple:we’llwritearecipethatwillcreateanexample.txtfileinyourhomedirectory.Therecipewecreatewillbecalledcreate_file,andwe’llputthatrecipeinsideacookbook,whichwillbenameddemo.

First,createthedirectorythatwillcontainthedemocookbook’srecipes(andanyinbetween):

user@host:~$mkdir-p~/chef/cookbooks/demo/recipes

Next,addthefollowingcodetoafile,create_file.rb,locatedinthedemocookbookdirectoryyoucreatedat~/chef/cookbooks/demo/recipes:

file"#{ENV['HOME']}/example.txt"do

action:create

content"Greetings#{ENV['USER']}!"

end

ThistellsChefthatwewanttocreateafile,$HOME/example.txt.ItscontentsshouldbeGreetings$USER,where$USERwillbereplacedwiththevalueof$USER,typicallytheloginnameofwhoeverisexecutingtherecipe.

TipForthoseunfamiliar,UNIX(andWindowsaswell)usesenvironmentvariablesasamechanismtoexchangedatabetweenprocesses.SomeenvironmentvariablesaresetwhentheuserlogsintothesystemsuchasHOME,USER,andavarietyofothers.ThesevariablesareavailableinRubyusingtheENVhash,wherethekeysarethevariablenames.InaUNIXshell,theseareaccessedusingthe$prefix.So,theuser’shomeisreferredtoas$HOMEintheshellandENV['HOME']insideRuby.

NowwewillneedtocreateaJSONdocumentthatdescribeswhatchef-soloshouldexecute.JSONisanacronymforJavaScriptObjectNotation,andChefusesJSONextensivelybecauseitiseasytoparse,humanreadable,andeasytogeneratefromallsortsoftoolsandlanguages.Createafile,node.json,locatedinourworkdirectory(~/chef/inthiscase)andaddthefollowingcontentinordertotellChefthatwewanttoexecutethenewlycreatedcreate_filerecipeinthedemocookbook:

{

"run_list":[

"recipe[demo::create_file]"

]

}

Here,wearedefiningthenodeashavingarunlist,whichisjustanarrayofthingstodo,andthattherunlistcontainsonerecipe,create_file,whichitcanfindinthedemocookbook(thegeneralformofarecipebeingcookbook::recipe).

Finally,we’lltellChefwheretofindthefileswejustcreatedusingasolo.rbfilethatwewillstoreinourworkingdirectory(~/chefinourcase):

CHEF_ROOT="#{ENV['HOME']}/chef"

file_cache_path"#{CHEF_ROOT}"

cookbook_path"#{CHEF_ROOT}/cookbooks"

json_attribs"#{CHEF_ROOT}/node.json"

Nowthatyouhavepopulatedtherequiredconfigurationfiles,youcanrunchef-soloandexecutetherunlistspecified.Inourcase,therunlistisdefinedasonlyonerecipe,create_file,butcanbeassimpleorascomplexasneeded.ThepreviousconfigurationtellsCheftoloadthenodeconfigurationfromthefilenode.jsontolookforcookbooksin~/chef/cookbooks/andtostoreanystatedatain~/chef/.Inordertoexecutethesecommands,youwillwanttorunchef-solo:

chef-solo-c~/chef/solo.rb

The-coptiontellschef-solowhichscriptcontainstheconfiguration.Onceyoudothis,youwillseetheactionsthatyourrecipeisperforming:

StartingChefClient,version11.8.2

CompilingCookbooks…

Converging1resources

Recipe:demo::create_file

*file[/home/user/example.txt]actioncreate

-createnewfile/home/user/example.txt

-updatecontentinfile/home/user/example.txtfromnonetob4a3cc

---/home/user/example.txt 2014-01-2023:59:54.692819000-0500

+++/tmp/.example.txt20140122-13411-1vxtg7v 2014-01-20

23:59:54.692819000-0500

@@-1+1,2@@

+Greetingsuser!

ChefClientfinished,1resourcesupdated

Onceitiscompleted,youwillseethat~/example.txtcontainsthegreetingthatyoudefinedintherecipe.Nowthatyouhavesuccessfullyusedchef-solo,let’smoveontotheChefservice.

InstallingaChefserverIfyourteamneedstohavecentralizedinfrastructuremanagementanddoesnotwanttouseahostedplatform,thenaself-installedChefserverisaperfectfit.ThisinstallationguideassumesthatyouwillberunningtheChefserveronasupportedLinux-basedsystem.

TheChefservicecomponentscanbeinstalledonasinglemachinewithoutanyissue.Installingitonasinglehostwilllimityourabilitytoscaleorbehighlyavailable,butwillprovideaverysimplepathtogettingstartedwiththeChefservice.

RequirementsandrecentchangesSincetheChefserviceisdesignedtobeamultiuserplatformandprovidesfunctionalitiesthatchef-solodoesnotoffer,theinstallationismorecomplexandinvolvesmoresoftwaretoachievethisfunctionality.ServicessuchasSolrforfull-textindexingandPostgreSQLfordatastoragecanbeasignificantresourceforconsumers,soyouwillwanttoinstallChefonahostwithsufficientmemoryanddiskspace.Asystemwith2GBofmemoryand5-10GBofdiskspaceavailablewillbeplentyforasmalltomediumsizedinstallation.Youwillneedmoreresourcesasyourrequirementsfordatastorageandindexingincreaseovertime,soplanaccordingly.

Additionally,forthosewhohaveinstalledtheChefserverbefore,theinstallationpathhasbeengreatlysimplified.InadditiontoreplacingCouchDBwithPostgreSQLastheprimarydatastorageengine,thereisnowasingleomnibusinstallationpackageforChefthatinstallsalloftherequirementsforChefatasinglelocationsothatitoperatesinisolationanddoesnotrequiredependenciestobeinstalledseparately.

InstallationrequirementsInordertoinstalltheChefservice,youwillneedtohavethefollowing:

AsystemrunningasupportedLinuxvariant(64bitUbuntuLinux10.04through12.10or64bitRedHatEnterpriseLinux5or6)—thiscanbephysicalorvirtual.Ifyoudonothavethelocalresourcesforthis,AWSorRackSpacecloudserversaregoodoptions.Anetworkconnectiontothehostinordertodownloadtheinstaller.Administrativeprivileges(usingsudoordirectrootaccess)onthehostwhereyouareinstallingtheservices.Enoughfreespaceonthehosttoperformthedownloadandinstallation(minimum500MB,includingthedownload,but1GBto2GBispreferred).

WhatyouwillbeinstallingAttheendofthissection,youwillhaveafullyfunctionalChefserviceinstalledandreadytoworkwith.Beforeyougetstarted,let’slookatwhatyouwillbeinstallingonyoursystemsothatyouknowwhattoexpect.ThecomponentsthatmakeupaChefserviceareasfollows:

TheChefAPIserviceMessagequeue(AMQP)DatastorageSearchserviceWeb-basedmanagementconsole

TheChefAPIserviceisresponsiblefordeliveringrunlistsandreceivinginformationfromnodesaswellasprovidingawayforasystemadministratortoconfigurerecipes,runlists,databags,andthelike.Inordertogeneratethisdata,theAPIservicereliesonitspersistentdatastorageengine,inthiscasePostgreSQL,tostoreitsdata.TheoptiontosearchfordataisprovidedbytheSolrsearchengine,andRabbitMQisresponsibleforgluingthemalltogether.Together,thesecomponentsprovideChefwiththeabilitytodistribute,store,index,andmanageyourinfrastructure’sconfigurationdata.

GettingtheinstallerTheeasiestwaytoinstallChefisthroughasingledownloadablepackage,whichisprovidedforUbuntu10.04through12.10andRedHatEnterpriseLinuxVersions5and6.Thispackage,referredtoastheomnibusinstaller,containseverythingyouneedtogetaserverupandrunning.Youcanfinditonhttp://www.getchef.com/chef/install/.

Atthetimeofwritingthis,11.0.10isthelatestversionandistheonethatwillbeusedforthisbook.Thenewerversionofthe11.xseriesofChefshouldhaveaverysimilar,ifnotidentical,configuration.Notethattheseinstallersaresomewhatlarge,beingthattheycontainallofthedependenciesneeded.Forexample,theUbuntu12.10packageforChef11.0.10isapproximately200MBinsize.

TipAlthoughthesearetheofficiallysupporteddistributionsandreleases,itisentirelypossiblethattheseinstallerswillworkondifferentbutcompatibledistributions.Itmaybepossible,forexample,touseCentOSinsteadofRedHatEnterpriseLinuxorDebianinsteadofUbuntu.However,thesewillmostlikelyrequiresomemanualdependencyresolutionsandmaynotworkwithoutalotofeffort(andeventhen,possiblynotatall).

InstallationoutlineInstallationonallsupportedplatformsisrelativelysimilar.TheonlykeydifferencesarethenamesofthepackagefilesthatyouwilldownloadandthecommandsyouwillusetoinstallChef.

Thehigh-levelstepsyouwilltakeareasfollows:

1. DownloadingtheChefinstallerforyourplatform.2. Installingthepackageasanadministrativeuser.3. ConfiguringtheChefservice.4. Testingtheserverusingcommand-linetools.

Becausesteps3and4willbethesameforbothUbuntuandRedHatinstallationprocedures,theinstructionswillbeinasectionfollowingtheRedHatinstallationguide.

InstallingonUbuntuThefollowingareinstructionsforanUbuntu-basedsystem;theywereperformedonanUbuntu12.04host,butshouldbeidenticalforallsupportedUbuntudistributions.ForRedHat-basedinstallationinstructions,seethenextsection.

DownloadingthepackageYoucandownloadthepackagebyreturningtothedownloadpagereferencedpreviously(http://www.getchef.com/chef/install/),oryoucandownloadVersion11.0.10directlyfromhttps://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-1.ubuntu.12.04_amd64.deb.

InstallingthepackageInordertoperformtheinstallation,openaterminalonyourUbuntuhost(eitherlocallyorconnectviaSSH)asauserwhohasadministrativeprivileges.Thiscanbedonedirectlyeitherastherootoranyuserwhohaspermissiontoexecutearbitrarycommandsviasudo.

Onceyoulogintothehost,navigatetowhereyouwanttostorethepackage(rememberit’squitelarge,approximately200MB)anddownloadthefileusingcurl:

user@ubuntu:~$curl-Ohttps://opscode-omnibus-

packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-

1.ubuntu.12.04_amd64.deb

Oncethefileisdownloaded,thedpkgtoolwillbeusedtoperformthepackageinstallation:

user@ubuntu:~$sudodpkg-ichef-server_11.0.10-1.ubuntu.12.04_amd64.deb

Oncethisisfinished,theUbuntu-specificportionofthesetupiscomplete,andyouwillneedtoconfigureChefusingthechef-server-ctlcommand,whichwewilldiscussintheConfiguringChefServersection,followingtheInstallingonRedHatEnterpriseLinuxsection.

InstallingonRedHatEnterpriseLinuxInstallationonaRedHatEnterpriseLinuxdistributionisasstraightforwardasinstallinganyotherpackage.YoudownloadthepackagetothelocaldiskandinstallitusingRPMtools.

DownloadingthepackageYoucandownloadthelatestversionofthepackagebyreturningtothedownloadpagereferencedpreviously(http://www.getchef.com/chef/install/),oryoucandownloadVersion11.0.10directlyfromhttps://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm.

Inordertoperformtheinstallation,openaterminalonyourRedHathost(eitherlocallyorconnectviaSSH)asauserwhohasadministrativeprivileges.Thiscanbedonedirectlyeitherastherootoranyuserwhohaspermissiontoexecutearbitrarycommandsviasudo.

Onceyoulogintothehost,navigatetowhereyouwanttostorethepackage(rememberit’squitelarge,approximately200MB)anddownloadthefileusingcurl:

user@rhel:~$curl-Ohttps://opscode-omnibus-

packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm

Howlongthistakeswillvaryaccordingtotheavailablebandwidthbutshouldtakesomewherebetween5and20minutesonareasonablyfastconnection.

Oncethefileisdownloaded,therpmtoolwillbeusedtoperformthepackageinstallation:

user@rhel:~$sudorpm-ivhchef-server-11.0.10-1.el6.x86_64.rpm

Oncethisisfinished,theRedHat-specificportionofthesetupiscomplete,andyouwillneedtoconfigureChefusingthechef-server-ctlcommand,whichwewilldiscussinthefollowingsection.

ConfiguringaChefserverHistorically,installingChefrequiresmanualeditingofconfigurationfiles,choosingRabbitMQcredentials,installingCouchDB,andahandfulofothertasks.Now,withtheomnibusinstaller,allofthisistakencareofforyou.Ifyouhavebeenfollowingalong,yoursystemhastheChefserverandallofitsdependenciesinstalledonthesysteminthe/opt/chef-serverdirectory.

Includedwiththeinstallationofthepackageisashellscript,chef-server-ctl(locatedat/opt/chef-server/bin),whichisresponsibleforconfiguringyournewlyinstalledChefserver.Inordertoconfigureyourservices,youwillneedtorunitasrootbecausethescriptswillneedtomodifyyoursysteminwaysthatyourregularaccountmaynotbeableto.Initializingtheconfigurationtoolisassimpleasissuingthefollowingcommand:

sudochef-server-ctlreconfigure

Runningthisscriptmaytakeafewminutes,anditwillproducealotofoutputwhileitisdoingitswork.Whileitisrunning,let’stakeafewminutestodiscusshowitworksandwhatitisdoing.

Understandinghowchef-server-ctlworksEarlierinthischapter,youwerebrieflyintroducedtothechef-solotool.Yousawhowitcanbeusedtomanageyourlocalserverusingon-diskrecipesandconfigurationdata.TheChefteamhasleveragedthisabilitytodojustthatwiththeChefserverusingchef-solotobootstraptheserverconfiguration.Ifyouweretolookatthecodeforthe/opt/chef-server/bin/chef-server-ctlscript,youwouldseethatthelastlineinthescriptexecutesthefollowingcommand:

/opt/chef-server/embedded/bin/omnibus-ctlchef-server/opt/chef-

server/embedded/service/omnibus-ctl$@

Ifyoufollowthetrailanddigintotheomnibus-ctlscript,youwillfindthatitisjustawrapperaroundtheomnibus-ctlRubygem.Diggingintotheomnibus-ctlgem,youwillseethatintheend,thereconfigurecommandyoupassonthecommandlineisaRubymethodthatmakesthefollowingcall:

run_command("chef-solo-c#{base_path}/embedded/cookbooks/solo.rb-j#

{base_path}/embedded/cookbooks/dna.json")

ThistellsusthattheChefomnibuspackageuseschef-solotoconfigureitself—aprettyclevertrickindeed!Youcanseejusthowpowerfulatoolchef-solocanbe,beingabletoconfigureandreconfiguretheChefservice.

What’shappeningonmyserver?Whatyouwillprobablynoticerightawayisthatalotoftextisbeingscrolledpastinyourterminalwindow.Ifyouweretolookatthecontents,youwouldseethatitshowsyoutheactionsthatarebeingtakenbychef-solotoprovisionyournewservices.Asthereisalotofinformationgoingpast(thousandsoflines),hereisahigh-leveloverviewofwhatishappeningonyourhost:

1. Anewuser,chef_server,anditscorrespondinggrouparebeingprovisioned.2. Chefservicesarebeingsetup,andstartupscriptsforupstartarebeingplacedinthe

appropriatesystemdirectories.TheRunscriptsforChefservicesarelocatedat/opt/chef-server/sv.

3. Chefstatedirectoriesarebeingcreatedin/varincluding/var/opt/chef-serverand/var/log/chef-server.

4. RabbitMQisbeingconfiguredtostoredatain/var/opt/chef-serverandlogtheoutputto/var/log/chef-serveraswellasitsstartupscriptsin/opt/chef-server/sv/rabbitmq/run.

5. PostgreSQLisbeingconfiguredwithitsdatain/var/opt/chef-server/postgresql/dataalongwithauser,opscode-pgsql,toruntheservice.Somesystem-levelchangestosharememorysizesarebeingsetviasysctltomakePostgreSQLworkaswellaspersistedinsystctl.conf.

6. Solrisbeingsetuptoworkwiththeconfigurationanddatarootedin/var/opt/chef-server/chef-solr/,withtherunscriptbeingplacedin/opt/chef-server/sv/chef-solr/run.

7. Chef-expander(thedata-indexingservice)isbeingconfiguredfor/var/opt/chef-server/chef-expanderasitsworkingdirectorywithSolrandRabbitMQendpointsonthelocalhost.Therunscriptislocatedat/opt/chef-server/sv/chef-expander/run.

8. TheChefbookshelfmetadataserviceisreadiedin/var/opt/chef-server/bookshelf/withitsrunscriptat/opt/chef-server/sv/bookshelf/run.

9. Erchef,theErlangChefservice,isinstalledandpointedatthelocalSolr,RabbitMQ,bookshelf,andPostgreSQLservices.

10. Thesystemisthenbootstrappedusingthebootstraprecipe.Thisrecipeverifiesthatthesystemisrunning(bycheckingthatthehttp://localhost:8000/_statusreturnsanHTTP200response)andinstallstheSSLcertificatefortheweb-basedUIin/etc/chef-server/chef-webui.pem.

11. Theweb-basedUIconfigurationfilesaregeneratedandplacedin/var/opt/chef-server/chef-server-webui/.

12. AcopyofnginxtohostthewebUIisplacedin/var/opt/chef-server/nginx,andtheinitialself-signedSSLcertificatesaswellasthestaticassetsareinstalledin/var/opt/chef-server/nginx/html.

13. TheChefAPItestingframework,chef-pedant,isinstalled.14. Finally,/etc/chef-server/chef-server-running.jsonisgeneratedwiththe

currentconfigurationsettingsforyourChefservices.

Clearly,thereisalothappeninghere;ifyouhaveanyoutstandingconcernsaboutwhatisbeingdone,besuretoreadthroughtheoutput.OneofthegreatthingsaboutChefisthattherecipesarejustasetofscriptsthatyoucanopenandviewthecontentsof,andtheoutputshowsyouwhatishappeningduringtheexecution.Everythingitdoesistransparentandmanageablebyyou.

VerifyingthattheservicesarerunningOncetheconfigurationofyourservicesiscomplete,youwillwanttovalidatethattherequiredservicesarerunning.Again,thechef-server-ctlscriptwillbeused,butwewillinvokethestatussubcommandinsteadofthereconfiguresubcommand,asshowninthefollowingcode:

user@host:~$sudochef-server-ctlstatus

run:bookshelf:(pid3901)3123s;run:log:(pid3900)3123s

run:chef-expander:(pid3861)3129s;run:log:(pid3860)3129s

run:chef-server-webui:(pid4053)3095s;run:log:(pid4052)3095s

run:chef-solr:(pid3819)3135s;run:log:(pid3818)3135s

run:erchef:(pid4230)3062s;run:log:(pid3937)3117s

run:nginx:(pid4214)3064s;run:log:(pid4213)3064s

run:postgresql:(pid3729)3146s;run:log:(pid3728)3146s

run:rabbitmq:(pid3423)3172s;run:log:(pid3422)3172s

ThestatussubcommandwillshowyoutheprocessIDofeachcomponent,howlongithasbeenrunningfor,thePIDoftheloggingprocessassociatedwiththatservice,andhowlongtheloggingservicehasbeenrunning.Forexample,wecanseethatchef-server-webuihasaPIDof4053andhasbeenrunningforclosetoanhour,andtheloggerhasaPIDof4052,havingbeenrunningforjustaslongastheservice.

Asyoucansee,theinstallationofChefyieldsanumberofcomponentsthatwillneedtobeupandrunninginordertosuccessfullyuseChef.Youshouldhavethefollowingcomponentsrunningandlisteningonthefollowingnetworkports:

Component Whattolookforintheprocesslist Port(s) Public?

ChefAPIserver Erchefandnginx 80,443 Yes

Webmanagementconsole chef-server-webuiandnginx 80,443 Yes

Dataindexer chef-expander N/A N/A

Solr java(runningstart.jarintheChefdirectory) 8,983 No

PostgreSQL postmaster 5,432 No

RabbitMQ beam.smprunningrabbit 5,672 No

Publiccomponentsneedtobemadeavailabletoanyclients,nodes,orendusersthatexpecttousetheChefserviceoverthenetwork.Configuringyourinfrastructuretoensurethatyourservicesareavailableviathenetworkisoutsideofthescopeofthisbookasthereareanear-infinitenumberofpossiblenetworkconfigurations.

Atahigherlevel,makesurethatanyfirewalldevicesorpacket-filteringsystemsarenotpreventingtrafficfromreachingtheseservicesifyouseethattheyarerunning,butarehavingdifficultiesinconnectingtothem.Ifanyoftheseservicesarenotrunning,youwillneedtoconsultthelogfilesgeneratedbytheservicetodeterminewhatmightbepreventingthemfromstartingup.

ValidatingthatyourserviceisworkingInordertoworkwithChef,youwillneedawaytointeractwithit.Fortunately,Chefprovidesasuiteofcommand-lineutilities,whichwewilldiscussatlengthasthebookprogresses.Thereisoneprimarytool,knife,thatallowsanadministratortointeractwiththeserviceinthecommandline.Theknifetoolisrunfromaworkstationandprovidesmanycommandstoview,search,andmodifydatamaintainedbytheChefservice.Onceyouhaveinstalledandverifiedthatalltheservicesarerunning,wecanmoveontosettingupknife.

TipYouwillseethatthestandardplacetostoreyourChefconfigurationdataisin$HOME/.chef(onaUNIX-likesystem.)Thisisnotmandatory,andthesefilescanbestoredanywhereyoulike.

TheknifetoolcommunicateswiththeChefserverviaHTTPandusescertificatesforauthenticationbetweentheworkstationandtheserver.Inordertogetstartedwithknife,wewillneedtodotwothings:gainaccesstothecertificatesthatweregeneratedduringtheinstallationofChefandthenusethosecredentialstosetupanewuserinthesystem.

Inthefollowingexamples,wewillbeusingthehostthattheChefserviceswereinstalledonasourworkstation(wherewewilluseknife).Ifyouwanttouseadifferenthost,youwillneedtogettherequiredcertificate(.pem)filestoyourlocalmachineusingscporsomeothermechanism.Byusingthefollowingcommands,wecangettherequiredauthenticationmaterialsintoourworkdirectory:

mkdir$HOME/.chef

sudocp/etc/chef-server/admin.pem$HOME/.chef

sudocp/etc/chef-server/chef-validator.pem$HOME/.chef

sudocp/etc/chef-server/chef-webui.pem$HOME/.chef

sudochown–R$UID$HOME/.chef

TipChefusesasignedheaderauthenticationforrequeststotheAPI,whichmeanstheremustbeasharedkeythatispresentonboththeclientandtheserver.Chef-serverwillgeneratethechef-validator.pemfilewhenitisconfigured.Newnodesorclientsusethechef-validator.pemfiletosigntherequestsusedtoregisterthemselveswiththesystem.

OnceyouhavethesefilescopiedintoyourChefworkdirectory,itistimetoconfigureknifeitself.Fortunately,knifehasaninteractiveconfigurationmodethatwillwalkyouthroughtheprocessofgeneratingaconfigurationfile.First,ensurethatyouareusingyourChefgemset(ifyouareusingRVMaswediscussedearlier)andthenrunknifeonyourworkstation(again,inthisexample,weareusingourChefservicehostforbothpurposes):

user@chef:~$rvmuse1.9.3@chef

user@chef:~$knifeconfigure-i

Whenyourunknifewiththe-iflag,youwillbepromptedbythefollowingquestions,

whichyoucananswerwiththedefaultsforalmosteverything(non-defaultanswersareinbold):

WARNING:Noknifeconfigurationfilefound

WhereshouldIputtheconfigfile?[/home/user/.chef/knife.rb]

PleaseenterthechefserverURL:[https://localhost:443]

Pleaseenteranameforthenewuser:[user]

Pleaseentertheexistingadminname:[admin]

Pleaseenterthelocationoftheexistingadmin'sprivatekey:[/etc/chef-

server/admin.pem]~/.chef/admin.pem

Pleaseenterthevalidationclientname:[chef-validator]

Pleaseenterthelocationofthevalidationkey:[/etc/chef-server/chef-

validator.pem]~/.chef/chef-validator.pem

Pleaseenterthepathtoachefrepository(orleaveblank):

CreatinginitialAPIuser…

Pleaseenterapasswordforthenewuser:

Createduser[user]

Configurationfilewrittento/home/user/.chef/knife.rb

user@chef:~$

Asmentionedearlier,thisdoestwothings:

First,itusesthevalidationkeyandclientnamespecifiedatthepromptstocontacttheAPIserviceandregisteranewclient(user)withtheserviceSecondly,itgeneratesaconfigurationfileforknifethathasthesettingsneededtoconnecttotheservicefromnowon

SinceChefanditscomponentsarewritteninRuby,theresultingconfigurationfileisaRubyscript,whichcontainssomecodethatconfiguresknifesothatitknowswhatAPIservertoconnectto,whichkeyfilestouse,whatclientnametouse,andsoon.

Aninspectionoftheconfigurationfilethatwasgeneratedbythepreviouscommandwilllooklikethefollowing:

log_level:info

log_locationSTDOUT

node_name'user'

client_key'/home/user/.chef/user.pem'

validation_client_name'chef-validator'

validation_key'/home/user/.chef/chef-validator.pem'

chef_server_url'https://localhost:443'

syntax_check_cache_path'/home/user/.chef/syntax_check_cache'

Becauseweareusingtheservicehostasourworkstation,theChefserverURLpointstothelocalhost.Ifyourworkstationweretobeadifferentsystemsuchasyourlaptop,thenthisURLwouldbetheIPorhostnameofthehostrunningtheChefservice.

EnsuringthatyourknifeconfigurationworksAftersettingupknife,wecanuseittovalidatethatitwasconfiguredcorrectlybyqueryingtheChefserverusingsomesimplecommands.Theknifecommandsfollowtheformatknife<command><subcommand>,wherecommandiseitheraclient,configuration,cookbook,cookbooksite,databag,environment,exec,help,index,node,recipe,role,search,ssh,status,ortag.Subcommandswillvarywiththecommand,buttheytypicallyincludethingssuchasshow,create,list,anddelete(amongothers).

Astherewillinitiallybenonodes,cookbooks,recipes,roles,databags,andsuch,wewillquerythelistofclientsthattheserverknowsabout.Thisshouldbealistoftwoclients:chef-webui(asitisaconsumeroftheAPIitself)andchef-validator(withoutit,itwouldn’tbepossibletoregisteranewclient).

Theclientcommand,withthelistsubcommand,printsalistofclientsthattheserverknowsabout.Atthispoint,runningthecommandwouldlooklikethis:

user@chef:~$knifeclientlist

chef-validator

chef-webui

user@chef:~$

TipIfyoudonotgetthepreviousoutput,butgetanerrorinstead,youwillneedtogobackandmakesurethatallthepreviousstepsarecompletedandverified.

Onceyouknowthatitworks,youcanuseknifetointeractwiththeAPI.Unfortunately,wedonothavemuchdatainthesystemjustyet,butwecanusetheshowsubcommandinconjunctionwiththeclientcommandandaclientnametodisplaymoredetailedinformationaboutaclient:

user@chef:~$knifeclientshowchef-webui

admin:true

chef_type:client

json_class:Chef::ApiClient

name:chef-webui

public_key:-----BEGINPUBLICKEY-----

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAos5cQ1NxP7zKf1zRM33g

YeVyHNOO5NcICjSIvqQ5A37wwLfgtPLJQqboW7ZcNL3xYcKOlfYSEK7xha3ss8tT

A+XMifaFp3JsdheyPeIJir2bc9iltUUcbpw9PJ2aQKTBlFNx23A7ag+zBfxcDjbY

7RkdcziwB74ynd6e/K8c0JTRnA5NxoHkFc6v8a/itwujGwugWJXDQunWfCmAvjws

JgDOUu2aHOCVIVkc8it51Sc7Anx0YnCjNmdhz1xIo0MOVNOEmC9ypP0Z7mVv1C69

WWBOEvS9zimjXo4rxBwFmWkPEIG6yPQjhuNmFd69K14vZQtAsH07AZFRSS7HLWnZ

WQIDAQAB

-----ENDPUBLICKEY-----

validator:false

user@chef:~$

SummaryCongratulations!Ifyouhavegottenthisfar,younowhaveafullyfunctionalChefserviceandacopyofthecommand-lineutilities,includingchef-solo.Younowhavecoveredthefollowing:

UsingRVMInstallingchef-soloCreatingasimplerecipeRunningrecipeswithchef-soloInstallingtheChefserviceGettingstartedwiththeknifeutilityVerifyingthatyourChefserviceisoperatingcorrectly

NowthatyouareabletouseyourChefservice,wecanbegintoinvestigatehowtomodelourenvironmentwithChefandseewhatitcandoforus.

Chapter2.ModelingYourInfrastructureNowthatyouhavesetupyourserverorareusingthehostedoffering,let’sdiscusshowtomodelyourinfrastructurewithChef.Chefallowsyoutodothisusingbuildingblocksthatshouldbefamiliartoanysystemadministrator.

ThischapterwillcoverhowtomodelyourinfrastructurewithChef.Thiswillinvolvethefollowing:

LearningsometerminologiesandconceptsrelevanttoChefAnalyzingasimpleapplicationinfrastructureandseeinghowitcanbemodeledwithChefDecomposingourarchitectureintothevariouscomponentstobemodeledExamininghowdataisstoredandconfigurationsaregeneratedwithChefBootstrappingandprovisioninghostswithcloudproviderssuchasAWSandRackspaceCloud

GettingtoknowChefAswithanyothertoolorsystem,therearenewconceptsandterminologiestobelearned.Herearesometermsthatyoumayhaveseenorwillseeinthischapter:

Node:AnodeisasystemthatismanagedbyChef.Thesecanbeservers,desktopsystems,routers,andanythingelsethatiscapableofrunningtheChefclientandhasasupportedoperatingsystem.Workstation:AworkstationisaspecialnodethatisusedbyasystemadministratortointeractwiththeChefserverandwithnodes.Thisiswherethecommand-linetoolsareexecuted,specificallytheknifecommand-linetool.Bootstrap:ThisisaprocessofsettingupanodetobeusedasaChefclient.ThisinvolvesperforminganyworkrequiredtoinstallthedependenciesforChefaswellasChefitself.BootstrapScript:ThereareanumberofpossiblewaystoinstallChef,Ruby,othercorerequirements,aswellasanyadditionalconfigurationthatisneededforyourspecificsystems.Toprovidethislevelofflexibility,thebootstrapprocessisscripted;onWindows,thisisabatchfile.Recipe:Recipesprovidetheinstructionsrequiredtoachieveagoal,suchasinstallingasoftwarepackage,configuringafirewall,provisioningusers,installingprinters,ormanagingothersystemresources.ThesearewritteninRubyandexecutedonthenodesspecifiedbythesystemadministratorthroughtheChefconsole.Cookbook:Acookbookisacollectionofrecipes;typically,acookbookprovidesonespecificgroupofactionssuchasinstallingApacheorMySQL,providingChefresourcesforaspecificsoftwaretool,andsoon.Attributes:Variouscomponentsofthesystemhavetheirownattributesandpropertiesthatdescribehowthesoftwareistobeconfigured.Thesepropertiesaredefinedatvariouslevels,rangingfromnode-specificsettingstogeneraldefaultsforacookbookorarole.Role:Aroleisacollectionofrecipesandconfigurationdatathatdescribehowaresourceshouldbeconfiguredinordertoplaythatroleinyouroverallsystemarchitecture.ExamplesofrolesmightincludeMSSQLServers,ExchangeServers,IISServers,fileservers,andsoon.Aroledoesnotcontainanyknowledgeofresources(systems)toapplytheroleto,onlytheconfigurationdata.RunList:Arunlistisalistofrecipestobeappliedtoagivennodeinacertainorder.Arunlistcanbecomposedofzeroormorerolesorrecipes,andtheorderisimportantastherunlist’sitemsareexecutedintheorderspecified.Therefore,ifonerecipeisdependentupontheexecutionofanother,youneedtoensurethattheyruninthecorrectorder.Resource:Resourcesareawayofdescribingwhatarecipeisperforming.Someexamplesofresourceswouldincludefiles,directories,printers,users,packages,andsoforth.Aresourceisanabstractionofsomethingthatisconcretelyimplementedinaprovider.Provider:Aproviderisaconcreteimplementationofaresource.Forexample,auser

isagenericresource,butLDAPusersorActiveDirectoryusersareconcreteimplementationsofauserresource.Thetypeofproviderbeingselectedwilldependonsomefactors,suchastheplatform.Databags:Databagscontainshareddataaboutyourinfrastructure.Informationthatisnotspecifictoaroleoraresource,suchasfirewallrulesanduseraccounts,willbestoredindatabags.Thisisagoodplacetostoresystem-wideconfigurationdata.Environments:Environmentsprovidealevelofencapsulationforresources.Forexample,youmayhavetwoidenticalenvironments,onefortestingandoneforproduction.Eachofthesemayhavesimilarsetupsbutdifferentconfigurations,suchasdifferentIPaddressesandusers.

ModelingyourinfrastructureNowthatyou’remorefamiliarwithsomeofthetermsyouneedtoknow,let’stakealookatasamplemodelandmapittoChef’scomponents.Atahigherlevel,theapproachwewilltakeisasfollows:

1. Defineanoverviewofyourinfrastructurethatisdecomposedintorolestobeperformedwithinthemodel(webservers,firewalls,databaseservers,andsoon).

2. Collectordeveloprecipesthatdescribetheconfiguration,software,andactionstobeappliedforthoseroles.

3. BootstraphostswiththeChefclientsothattheycanparticipateinbeingmanaged.4. Addanyrequiredconfigurationdataintodatabagstobeusedbynodeswhenrunning

recipessuchasIPaddressranges,hostnames,users,softwarekeys,oranythingelsethatisspecifictotheactiveconfiguration.

5. Segregatehostsandconfigurationsintodifferentenvironmentstoprovideareplicatedinfrastructure(development,staging,production,andsoon).Thisstepisoptional.

Inthischapter,wewillbeusingCheftobuildtheinfrastructureforamulti-tiered,photo-sharingapplicationwhosecomponentsarediagrammedinthefollowingimage:

Buildinganarchitecturediagramgivesusagoodoverviewofoursystemsothatwecanhaveamapofthesystembeforewestartbuildingit.Itisimportanttonotethatamodelof

ourinfrastructuredoesn’tneedtobemappeddirectlytoresources(physicalorvirtual);rather,itprovidesanabstract,high-leveloverviewofyoursystems.Onceyouhavethemodel,youcanapplyittotheresourcesavailableasyouseefit.

Oursampleservice-orientedwebapplicationiscomposedofthefollowingsoftwareservices:

AfrontendwebapplicationserviceAnimage-processingengineAnimagesearchengine

Eachofthesecomponentsisarolethatisbeingplayedinthesystem.Theserolesmaycoexistonsharedresourcesormaybeappliedtodedicatedresources.Aservice-orientedarchitectureisagoodexampletoworkwithforseveralreasons:

ItisflexibleandscalableItwillprovideuswithacompletesystemthatiscomposedofmultipleindependentcomponentstomodel,makingitmoreinterestingasanexample

Inthisexample,inadditiontotheseroles,wemightwanttofurtherconfigureourinfrastructuretoprovidetwodifferentenvironments:oneforstagingandintegrationtestingandoneforproduction.Again,becausethisisamodel,ourstagingenvironmentandproductionenvironmentwillbecomposedofthesamerolesandhavethesameoverallarchitecture;however,eachwillhavedifferentresourcesandconfigurationdataassociatedwiththem.Youmaychoose,forexample,toconsolidateresourcesinatestenvironmentinordertokeepcostsdown.

Forthisinitialoverview,wewillassumethatwehaveanaccountwithapopularcloud-server-hostingcompany,thatthenetworkandoperatingsystemsareinstalledandoperational,andthatwehaveafunctionalandconfiguredChefserviceandworkstation.

Inourhypotheticalsystem,eachservicecanbemappedtoaspecificroleinChef.Tomodeltheinfrastructuredescribed,wewillhaveanumberofroles,oneperelementinourarchitecture.Inthiscase,wewillbuildoneroleforeachserviceinourstackastheyprovideveryspecificfeatures.

RolesAroledescribesapartthatasystemplaysinyourinfrastructurethroughacombinationofrecipestoexecuteandconfiguredata.Theserolescanbefine-grainedorbroadlydescribed,dependingonyourneeds.Therearebenefitsanddrawbackstoboththeapproaches:fine-grainedrolesaresmallerandeasiertoworkwithbutrequirealargernumberofrolestomanage,whereasbroadlyscopedrolesarelessflexibleandnotasreusable.

Forexample,consideratypicalLAMP(Linux,Apache,MySQL,andPHP)stack.Thestackcouldberepresentedbythreeroles:anApachewebservicewithPHP,aMySQLdatabaseservice,andanOpenSSHserviceforadministration.Alternatively,youcoulddefineonerolethatdescribestheinstallationoftheMySQLdatabaseservice,theSSHservice,andtheApacheservice.

Rolesthemselvesknownothingaboutresources;instead,theyareadescriptionofhowtoconfigureasysteminordertofillthatrole.Thesystemadministrator,viathechefconsole,assignsrolestothenode(s)thattheywillbeappliedto.Thismaybeaone-to-one,one-to-many,ormany-to-onemapping,dependinguponyourcapacityplanning.Atanytime,anadministratorcanchangethelistofrolesthatareappliedtoanode,addingorremovingthemasneeded.Forexample,youmightdecidetoapplyallyourrolestoonehosttodayforcostsavings,butscalethemoutinthefutureasyourbudgetandneedsgrow.

DefiningrolesLet’stakealookatsomeroleswemightdefinetomodelourSOAapplicationonasdescribedearlierinthechapter.Here,wewilldefinefine-grainedrolesastheyareeasiertodissectanddeployontoseparatenodeslater.Atahigherlevel,thefollowingrolesarewhatourservicesneedtoprovide.

Awebapplicationservicerole

Whendefiningwhatawebapplicationserverwillneedtodo,wewillneedthefollowing:

nginxHTTPserviceRuby2.0MemcachedservicePostgreSQLclientlibrariesOpenTCPportsontheexternalnetworks:80forHTTPand443forHTTPS

Animage-processingrole

Thisrolerequiressomeimage-processinglibrariesandcustomsoftwaretobeinstalled:

ImageMagicklibrariesGit(tocheckoutthesourcecode)Buildtools(tocompileoursource)Thelatestversionofourimage-processingsoftware

Animagesearchrole

Aservicethatprovidesimagesearchingthroughperceptualhashingwillprovideanimagesearchrolefunctionality.Thisrolewillrequirethefollowing:

AJavaruntimeenvironment(JREorJDK)Ourcustom-builtservicethatisdevelopedinJavaTCPport8999opentointernalhosts

APostgreSQLservicerole

ForthePostgreSQLdatabaseservicerole,thelistisasfollows:

PostgreSQL9.xserverTCPport5432opentointernalnetworkclientsDatabasebackupsoftwaretobackupdatatoanexternalclouddatastorageservicesuchasS3

ASolrservicerole

AsystemthatprovidestheApacheSolrservicewillneedthefollowing:

AcompatibleJavaruntime(OracleJREorOpenJDK)TCPport8993opentointernalserversApacheSolritself

AnOpenSSHservicerole

AnOpenSSHservicerolewillneedthefollowing:

OpenSSHserverTCPport22openonalltheinterfaces

Noticethattheseroleshavenospecifichostinformation,suchasIPaddressesorserverstoinstallthesoftwareonto;instead,theyareblueprintsforthepackagesweneedtoinstallandtheconfigurationthatthoseroleswillprovide,suchasopenports.Inorderfortheseroledefinitionstobemadeasreusableaspossible,wewillwriteourrecipestousenode-androle-specificconfigurationordatafromourdatabagstoprovidetherequiredconfigurationdata.

Inordertodefinetheseroles,youwillneedrecipesthatdescribethesetsofstepsandoperationsthatwillbeappliedtohostsinordertofulfilleachrole.Forexample,thePostgreSQLdatabaseserverwillrequireyoutoinstallPostgreSQL,openthefirewall,andsoon.Thesedefinitionsarecreatedbydevelopingrecipesthatcontainthenecessaryinformationtoperformthetasksrequired,suchasinstallingpackages,generatingconfigurationfiles,executingcommands,andsoon.Mostoftheservicesmentionedhere(ourcustomimagingsoftwarebeingthelikelyexception)havecookbooksthatalreadyexistandareavailablefordownload.

ImplementingaroleNowthatyouhaveseenwhatourinfrastructuremightlooklikeatahigherlevel,let’stakealookathowwewillgoaboutimplementingoneofourrolesinChef.Here,wewillimplementthePostgreSQLserverroleasitissimpletoconfigureandhasaveryrobustcookbookavailablealready.

Asmentionedbefore,youwillneedtoeitherdevelopyourowncookbooksordownloadexistingonesinordertobuildyoursystems.Fortunately,therearethousandsofcookbooksalreadywritten(over1,500asofthiswritingintheChefSupermarket)and,aswewillseeinfurtherchapters,developingnewcookbooksisastraightforwardprocess.

Inordertodefinearole,weneedtocreateit;thiscanbeaccomplishedthroughawebinterfaceorbyusingknife.Here,andelsewhereinthisbook,wewilluseknifeasthewaytointeractwiththeChefservicebecauseitprovidesaconsistentexperienceacrossself-managedandhostedChef.Solet’sgetstarted!

Thefirstthingyouwillneedtodoiscreateanewrolewithknife,whichisassimpleasexecutingthefollowing:

kniferolecreate-dpostgresql_server

ThiswilltellknifetoconnecttoyourChefserver’sAPIandcreateanewrolenamedpostgresql_server.The-dflagtellsknifetoskipopeninganeditorandinsteadacceptthedefaultvalues.IfyouwanttoseewhattheunderlyingJSONlookslike,omitthe-dflagandmakesureyouhaveanappropriateEDITORenvironmentvariableset.Onceyourunthis,youcanverifythatyourrolewascreatedwiththefollowingcommand:

kniferolelist

Thiswillshowyouthatyouhaveasingleroleinthesystem,postgresql_server.Currently,however,thisroleisemptyandhasnoinformationassociatedwithit,justanameandanentryinthesystem.Nowthatwehavearoleinthesystem,let’slookathowwecanworkwithsomerecipestomakeourroledosomethinguseful,suchasinstallthePostgreSQLservice.

DeterminingwhichrecipesyouneedRecipesarehowChefknowshowtomakesurethatthecorrectpackagesareinstalled,whatcommandsneedtobeexecutedinordertoopenportsonthefirewall,whichportsneedtobeopened,andsoon.Likeanygoodcook,Chefhasawidearrayofcookbooksatitsdisposal,eachofwhichcontainsrecipesrelevanttothatparticularsetoffunctionality.ThesecookbookscaneitherbedevelopedbythesystemadministratorordownloadedfromavarietyofplacessuchasGitHub,BitBucket,orfromacollectionofcookbooksmaintainedbytheChefcommunityontheChefSupermarket(http://supermarket.getchef.com).Wewilldiscusshowtodownloadandgetstartedwithsomesimplerecipesandthenfurtherdiscusshowtodevelopanddistributeourownrecipesinlaterchapters.

Consideringhowwehavearrangedourroles,wewouldneedrecipestoinstallandconfigurethefollowing:

nginxAPostgreSQLserverAPostgreSQLclientRuby2.0SolrJavaOpenSSHAMemcachedserverMemcachedclientlibrariesImageMagickGitACustomimagingsoftware(wewillcallitImage-O-Rama)

Here,wewilltakeanin-depthlookatthereciperequiredforourPostgreSQLserverandhowwecanleveragethattoinstalltheserviceonahost.

InstallingacookbookInstallingacookbookforuseonourclientsisquitesimpleandinvolvesonlytwosteps:

1. Developingacookbook,ordownloadingthecookbookfromsomewhere.2. UploadingthecookbooktotheChefserviceusingknife.

Togetstarted,wewilldownloadanexistingPostgreSQLcookbookfromtheChefcookbookcollectionanduploadittoourChefservice.NotethatinordertoinstallthePostgreSQLcookbook,youwillalsoneedtoinstallanydependenciesthatarerequired.Forsimplicity,theyareprovidedhereaspartoftheinstructions;however,youmayfindthatwhenyouexperimentwithothercookbooksinthefuture,youwillneedtodownloadafewcookbooksbeforeallofthedependenciesaremet,oruseatoolsuchasBerkshelfformanagingthem.

TodownloadacookbookfromChef’sprovidedcollectionofcookbooks,wewilluseknife

withthefollowingcommand:

knifecookbooksitedownload<cookbook_name>

Inthiscase,wewillneedtodownloadfivedifferentcookbooks:

postgresqlbuild-essentialaptchef-sugaropenssl

Foreachoftheitemsinthelist,wewilldownloadthemusingthefollowingcommand:

knifecookbooksitedownloadpostgresql

knifecookbooksitedownloadbuild-essential

knifecookbooksitedownloadapt

knifecookbooksitedownloadchef-sugar

knifecookbooksitedownloadopenssl

Eachdownloadwillresultinanarchivebeingdownloadedtoyourworkstation.Thesearchivescontainthecookbooks,andyouwillwanttodecompressthemafterdownloadingthem.Theycanbedownloadedanywhere,butitwouldprobablybeagoodideatokeeptheminacommoncookbooksdirectory,somethinglikechef/cookbooksinsideyourhomedirectorywouldbeagoodideaifyouneedone.

Oncetheyaredownloadedanddecompressed,youwillneedtouploadthemtotheChefservice.Thiscanbedonewithonlyonecommandusingknifecookbookuploadasfollows;theyareuploadedfromthedirectoryinwhichyoustoredyourdecompressedcookbooks:

knifecookbookupload-o.aptbuild-essentialpostgresqlchef-sugar

openssl

Thiswilluploadthefivecookbookswedownloadedandtellknifetosearchthecurrentdirectorybywayofthe-o.directive.Oncethisisdoneyoucanverifythattheyhavebeeninstalledusingtheknifecookbooklistcommand.

Oncetheyareinstalled,yourcookbooksareregisteredwiththeChefservice,andwecantakealookathowwecanconfigureandapplythePostgreSQLservertoanewUbuntuhost.

ApplyingrecipestorolesNowthatyouhavesomecookbooksregisteredwithyourChefservice,youneedtoaddthemtoarole’srunlistinorderfortheirbehaviortotakeeffectonanyendhosts.Therelationshipbetweenarecipeandanygivennodeisshowninthefollowingdiagram:

Becauseofthenatureofthisrelationship,recipesdeliberatelyhavenoknowledgeofindividualnodes.Justasarecipeforchocolatechipcookieshasnoideaaboutwhomanufacturedtherollingpinandspatula;aChefrecipeissimplyasetofinstructionsonwhattodoandinwhatordertoperformthoseactions.

Becausewehaveuploadedourcookbookstothesystem,wehavealreadyaddedtherecipescontainedinsideofthosecookbookstooursystem;therefore,wecannowassociatearecipewithourrecentlycreatedrole.Ifyoulookatthecontentsoftherecipesdirectoryinsideofthepostgresqlcookbook,youwillseethatthereisaserver.rbfile.ThisdescribesarecipetoinstallthePostgreSQLserverandiswhatwewillbeaddingtoourpostgresql_serverroleinordertoperformtheactualinstallation.

Todothis,weneedtoeditourroleandaddtherecipetoitsrunlist;wewilldothisusingknife.

TipEnsurethatyouhaveavalidtexteditorinyourEDITORenvironmentvariable;otherwise,youwillhavedifficultyeditingyourentitieswithknife.

Inordertoeditourrole,wecanusethekniferoleeditcommand:

kniferoleeditpostgresql_server

ThiswillopentheJSONfilethatrepresentsthepostgresql_serverrolestoredintheChefserverinatexteditorwhereyoushouldseethefollowingcontent:

{

"name":"postgresql_server",

"description":"",

"json_class":"Chef::Role",

"default_attributes":{

},

"override_attributes":{

},

"chef_type":"role",

"run_list":[

],

"env_run_lists":{

}

}

ThemostimportantsectionofthisJSONblobatthismomentistherun_listkey—thisisanarrayofallthethingswewanttorun.Thiscanbealistofrecipesorroles,andeachofthosehasthefollowingnamingstructure:

recipe[cookbook::recipe]forrecipesrole[role_name]forroles

Soourserverrecipeinsideourpostgresqlcookbookwouldthereforebenamed"recipe[postgresql::server]".Thisisexactlywhatwewillbeaddingtoourrole’srunlistJSON.Updatetherun_listentryfromtheoriginalvalueofanemptyarray:

"run_list":[

],

ToincludeourPostgreSQLserverrecipe,usethefollowingcode:

"run_list":[

"recipe[postgresql::server]"

],

ThisisallweneedtochangenowinordertoapplythePostgreSQLserverroletoournode.

TipNoticethatwehavenotaddedanyvaluestotherole’sattributes;thismeansthatourrecipewillbeexecutedusingitsdefaultattributes.Mostrecipesarewrittenwithsomesetofacceptabledefaultvalues,andthePostgreSQLserverrecipeisnodifferent.

Fornow,thereisnoneedtomodifyanythingelse,sosavetheJSONfileandexityoureditor.DoingthiswilltriggerknifetouploadyourmodifiedJSONinplaceofthepreviousJSONvalue(afterdoingsomevalidationonyourJSONcontents),andtherolewillnowhavethepostgresql::serverrecipeinitsrunlist.Youshouldseeanoutputfromknifeindicatingthattherolewassaved,andyoucanverifythatthisisthecasewithasimplekniferoleshowcommand:

kniferoleshowpostgresql_server

Thiswillshowyouanoverviewoftheroleinamorehuman-readableformatthanthesourceJSON.Forexample,ourroleshouldnowhaveoneentryintherunlistasshowninthefollowingoutput:

chef_type:role

default_attributes:

description:

env_run_lists:

json_class:Chef::Role

name:postgresql_server

override_attributes:

run_list:recipe[postgresql::server]

Oncethisiscomplete,ourroleisnowreadytobeappliedtooneofournodes.Atthispoint,wehaveuploadedourcookbooks,definedarole,andassociatedarecipewithournewlycreatedrole.Nowlet’stakealookatthefinalstep:applyingournewroletoanode.

MappingyourrolestonodesAshasbeendiscussed,rolesareadefinitionofwhatcomponentsneedtobebroughttogethertoserveaparticularpurpose;theyareindependentofthehardwaretheyareappliedto.Thishelpstoseparateconcernsandbuildreusablecomponentstoacceleratetheconfigurationofinfrastructureinnewarrangements.Inordertomanifestarole,itmusthaveanodethattheroleisappliedto;inordertomanageanode,itmusthavetheChefclientanditsdependenciesinstalledandberegisteredwiththeChefservice.

OnceanodeisregisteredwithChef,youcansetnode-specificproperties,assignrolesandrunthechef-clienttoolonthehostinordertoexecutethegeneratedrunlists.Foroursampleapplicationstack,wemayhavethefollowinghostsrunningUbuntuLinux14.04:

cayennewatermelonkiwi

OncetheyarebootstrappedandregisteredwiththeChefservice,wewillthendecidewhichrolesaretobeappliedtowhichnodes.Thiscouldyieldaconfigurationthatlookslikethefollowing:

cayenne

Webapplicationservicerole

watermelon

APostgreSQLdatabaseroleASolrsearchenginerole

kiwi

Animage-processingroleAnimagesearchrole

Withoutanyhardware,rolesarejustanabstractblueprintforwhatneedstobeconfiguredtogethertoprovideaparticulartypeoffunctionality.Here,wehavecombinedourresources(cloudinstancesordedicatedhardware)andourrecipestobuildaconcreteinstanceofourservicesandsoftware.

Inordertoapplyournewlycreatedroletoourhost,watermelon,wewillneedtobootstrapthathost,whichwillinstalltheChefclientonthehostandthenregisteritwiththeChefservice.Thisisreallyasimpleprocess,aswewillseehere,andisachievedusingtheknifebootstrapcommand:

knifebootstrap-xroot-dubuntu14.04<ipaddress>

TipForourexample,thenodewilluseanUbuntu14.04hostcreatedonDigitalOcean,aninexpensivecloud-hostingprovider;youcanbootstrapjustaboutanymodernLinuxdistribution,butifyouarefollowingalongwiththecommandsinthebook,youwillget

thebestresultsbyusinganUbuntu14.04machine.

ThisprocesswillgothroughthestepsrequiredtoinstalltheChefclientonthenodeandregisteritwithyourChefservice.Onceitiscomplete,youwillseethattheChefclienthasfinishedwithanoutputsimilartothefollowing:

ChefClientfinished,0/0resourcesupdatedin4.264559367seconds

Ifyouwanttoverifythatthehosthasbeenadded,asimpleknifenodelistcommandwillshowyouthatithasbeenregisteredwiththeChefservice.Ifyoudon’tseetheclientoutputabove,oryoudon’tseethenewlybootstrappednodeinyourlist,makesurethattheoutputofknifebootstrapdoesn’tindicatethatanythingwentwrongalongtheway.

Onceournodeisregistered,wecanaddourpostgresql_serverroletoournode’srunlistusingthefollowingknifecommand:

knifenoderun_listsetwatermelonrole[postgresql_server]

Thiscommandwillsettherunlistonournewhost,watermelon,tocontainthepostgresql_serverroleasitsonlyentry.Thiscanbeverifiedusingtheknifenodeshowcommand:

knifenodeshowwatermelon

NodeName:watermelon

Environment:_default

FQDN:watermelon

IP:162.243.132.34

RunList:role[postgresql_server]

Roles:

Recipes:

Platform:ubuntu14.04

Tags:

Nowthatthenodehasarunlistwithentries,it’stimetoactuallyconvergethenode.ConverginganodemeansthattheChefserverwillcompiletheconfigurationattributesandthenprovidetheendhostwithacompletelistofrecipestorunalongwiththerequiredcookbookdataandthenexecutethemonthenode.

ConverginganodeConverginganodeisdonebyexecutingthechef-clientcommand-lineutilityonthehost;thiscanbedoneinoneoftwodifferentways.ThesimplestwayistoSSHintothehostusinganSSHclientandthenexecutechef-clientastheroot;anotherwayistouseknifetoexecuteacommandonasetofhostsinparallel,whichwillbediscussedinlaterchapters.Fornow,simplySSHintoyourserverandrunchef-clientastheroot:

root@watermelon:~#chef-client

TheChefclientwillconnecttotheChefserviceanddownloadanyinformationneededtoexecuteitscompleterunlist.Anode’srunlistisdeterminedbyexpandingeveryentryinthenode’srunlistuntilitisalistofrecipestoexecute.Forexample,ournodecontainsoneelementinitsrunlist,thepostgresql_serverrole.Thisrole,inturn,containsoneentry,thepostgresql::serverrecipe,whichmeansthatthefullyexpandedrunlistforournodecontainsonlyoneentry.Inthissimplecase,wecouldhavejustaddedtherecipedirectlytoournode’srunlist.However,thishasanumberofshortcomings,includingnotbeingabletoaddextraconfigurationtoallthePostgreSQLserversinourinfrastructure,aswellasanumberofotherreasonsthatwillbediscussedinthefuture.

Inadditiontocomputingtherunlist,theChefservicewillalsodeterminewhatthefinalsetofconfigurationdataforournodewilllooklikeandthendeliverittotheclient.Thisiscomputedaccordingtoasetofrulesshownlaterinthischapter.Oncethatisdelivered,theclientwillalsodownloadallthecookbooksneededinordertoknowhowtoexecutetherecipesspecifiedinthefinalrunlist.Inourcase,itwilldownloadallthefivecookbooksthatweuploadedpreviously,andthen,whentheclientruniscomplete,theresultwillbepresentedinarunningPostgreSQLserver.

Oncetheclientruniscomplete,itwillreportonhowlongtheruntookandhowmanyresourcesweremodified.Theoutputshouldlooksomethinglikethefollowing:

ChefClientfinished,8/10resourcesupdatedin61.524995797seconds

Assumingthatnothingwentwrong,youshouldnowhaveafullyfunctionalPostgreSQLserverdeployedtoyournewhost.ThiscanbeverifiedbylookingattheprocesslistforaPostgreSQLprocess:

root@watermelon:~#psax|greppost

11994?S0:00/usr/lib/postgresql/9.3/bin/postgres-D

Thereyouhaveit,withonlyonecommand;yournodehasnowbeenprovisionedasaPostgreSQLdatabaseserver.Nowlet’stakealookathowwecanusesomeotherfeaturesofCheftomodelourinfrastructure.

EnvironmentsBeyondcreatingrolesandhavingresourcestoapplythemto,thereareoftenrequirementsaroundgroupingcertainresourcestogethertocreateadistinctenvironment.Anexampleofthismightincludebuildingastagingenvironmentthatfunctionsexactlylikeaproductionenvironmentforthepurposesofpreproductiontestingandsimulation.Inthesecases,wewouldhaveanidenticalsetofrolesbutwouldverylikelybeapplyingthemtoadifferentsetofnodesandconfigurationvalues.Chefachievesthisseparationthroughtheenvironmentprimitive,whichprovidesawaytogroupseparateconfigurationsandhoststogethersothatyoucanmodelsimilar,yetseparate,portionsofyourinfrastructure.

Inourhypotheticalinfrastructure,thethreehostsinourproductionenvironmentmaybecondenseddowntooneserverinpreproductioninordertosavemoneyonresources(orforotherreasons).Todothis,wewouldneedtobootstrapanode,perhapsnamedpassionfruitandthenconfigureittohavealloftherolesappliedtoit,ratherthanspreadingthemoutacrosssystems,asshowninthefollowingfigure:

Here,inthepreviousimage,youcanseethateachenvironmenthasaverysimilarsetupbutadifferentsetofIPaddressesandresources.Eventhoughwehaveaheterogeneoushardwarescaleinourenvironments(productionhasthreenodes,andpreproductionhasonlyone),anychangeswemakewillbeappliedtoallofthesystemsinaconsistentmanner.

Inordertoachievethistypeoffunctionality,Chefneedsawaytoorganizeandcompiletheconfigurationdatainordertoprovideittotheendhostwhenthetimecomestoconfigurethehost.NowthatweunderstandhowtomodeloursystemswithChef,let’stakealookathowChefhandlestheconfigurationdatatomakeallofthishappen.

OrganizingyourconfigurationdataChefrunsonconfigurationdata—thisdatacanbestoredinavarietyofdifferentlocations,andwithavarietyofpurposes.Whencomputingthefinalconfigurationforagivennode,thesourcesofconfigurationdataarethen“squashed”intoasingle,authoritativeconfigurationtobedeployed.Thosesourcesincludethefollowing:

Cookbooks:ToprovidereasonabledefaultsforrecipesNodes:Node-leveloverridesanddefaultsRoles:Per-roleconfigurationdataDatabags:System-wideconfigurationdatastorage

Datafromtheselocationsiscombinedtoproduceafinalhashofattributeswhenaclientrequestsitsrunlistfromtheserver.Cookbooksprovideabaselinesetofattributesthattherecipesinsiderelyon.Theseattributesactas“sanedefaults”fortherecipesthat,intheabsenceofoverridingvalues,aresufficienttoexecutetherecipeswithoutextrawork.Othersources,includingtheenvironment,roleandnodeitself,mayinturnoverridetheseattributesinordertoprovidethefinalconfiguration.

Whendevelopingrecipes,theseattributescanbeaccessedthroughthenodehashandarecomputedbyChefusingasetofrulestodetermineprecedence.Theorderofprecedencewhencomputingthishashisbrokendownintothefollowinglevels(lowesttohighestpriority):

DefaultNormal(alsoset)Override

Withineachlevel,thesourcesofattributedatainorderofincreasingprecedenceareasfollows:

TheattributesfileinsideofacookbookEnvironmentRoleNode

Thismeansthatanode-specificoverrideattributetakesprecedenceoverallothers,whichinturnismoreimportantthantherole,environmentandcookbookoverrideattributes,andsodownthechainofprecedence.Asyourscopebecomesnarrowedfromthemostgenericdescriptionofarole—therecipes—tothemostspecificcomponentinthesystem—theactualnodeitself—thesesettingsoverridethemoregenericvalues.Anodeknowsbestwhattheauthoritativeconfigurationshouldbeasopposedtoarecipe,whichdoesnotknowanythingaboutresourcesonthehost.Forexample,considerthefollowingscenarioinwhichyouhavetwohosts,potassiumandchromium.Forsomelegacyreason,theirdisksareconfiguredslightlydifferently,asfollows:

Potassium:

16GBrootpartition

250GBSSDdatapartitionin/opt

Chromium:

32GBrootpartition400GBEBSdiskmountedat/usr

InordertoinstallthePostgreSQLdatabaseserver,youneedtomakesureyouinstallitatalocationthatprovidesenoughstoragespaceforthedata.Inthisexample,therewillbemoredatathaneitherrootdiskscancontain.Asaresult,thedatadirectorywillneedtoresidein/optonpotassiumand/usronchromium.ThereisnowaythatthePostgreSQLrecipecanaccountforthis,andthepostgresql_serverrecipedoesnotknowanythingaboutitsresources.Subsequently,thelogicalplacetoconfigurethedatadirectoryisatthenodelevel.Ifthedefaultlocationaccordingtotherecipewere/usr/local,thenanode-leveloverridemaynotbeneededforchromium;however,inthecaseofpotassium,itcouldbedirectedtostoredatain/opt/datainstead.

Whatallthismeansisthatasyoudeveloprecipes,anydefaultattributesetbyyourcookbookwillbethelowestpriority.Youcansafelysetsomereasonabledefaultsinyourcookbookknowingthattheywillonlybeusedasafallbackifnobodyoverridesthemfurtherdownthechain.

ExampleattributedataAsimpledefaultattributesfileforPostgreSQLcookbookmightlooklikethefollowing:

default['postgresql']['port']="5432"

default['postgresql']['data_dir']="/usr/local/pg/data"

default['postgresql']['bind_address']="127.0.0.1"

NoticethattheattributesforacookbookareaRubyhash.Typically,goodpracticedictatesthatthenamespace(firstkeyinthehash)isthesamenameasthecookbook(inthiscase,postgresql),butthisdoesnotneedtobethecase.Duetocookbooksoftencontainingmultiplerecipes,acookbook’sattributesfilewilloftenhaveper-recipedefaultconfigurations.ConsiderafurtherevolutionofthePostgreSQLattributesfileifitweretocontainrecipesforboththeserverandtheclientinstallation:

default[:postgresql][:client][:use_ssl]=true

default[:postgresql][:server][:port]="5432"

default[:postgresql][:server][:log_dir]="/var/log/pglog"

Therearetimeswhenjustasimpleattributesfiledoesn’tmakesensebecausetheconfigurationmaybedependentonsomepropertyofthenodebeingmanaged.ThefactthattheattributesfileisjustaRubyscriptallowsustoimplementsomelogicinsideourconfiguration(thoughyoushouldavoidbeingoverlyclever).Considerarecipewherethedefaultgroupfortherootuserdependsontheplatformyouareusing:"userdonBSDs,"nBSDsonUbuntuLinux,and"nUbuelsewhere.Chefprovidesamethod,value_for_platform,thatallowstheattributetobechangeddependingontheplatformtherecipeisbeingexecutedon,asthefollowingexampledemonstrates:

default[:users][:root][:primary_group]=value_for_platform(

:openbsd=>{:default=>"wheel"},

:freebsd=>{:default=>"wheel"},

:ubuntu=>{:default=>"admin"},

:default=>"root"

Whereitmakessense,attributescanalsobesharedbetweencookbooks.Therearelimitedusesforthis,anditshouldbeusedwithcareasitblurstheboundariesbetweencookbooksandcausesthemtobecometootightlycoupledwithoneanother.

DatabagsTherearetimeswhenconfigurationdatatranscendsarecipe,role,environment,ornode.Thistypeofconfigurationdatatendstobesystem-widedatasuchasthefollowing:

FirewallrulesforvarioustypesofhostsUseraccountsSSHkeysIPaddresslists(whitelistsandblacklists)APIkeysSiteconfigurationdataAnythingthatisnotuniquetoaspecificentityinyourinfrastructure

Databagsareveryfree-form,asthenameimplies;recipesthatrelyondatafromdatabagswillimposetheirownexpectationsoftheorganizationwithinadatabag,butChefitselfdoesnot.Databagscanbeconsidered,likeallotherChefconfigurationdata,tobeonelargehashofconfigurationdatathatisaccessibletoalltherecipesacrossallthenodesinthesystem.

KnowingwhentousedatabagsBuildingfirewallrulesareagoodusecasefordatabags.Agoodcookbookisanislandbyitself;itmakesasfewassumptionsabouttheworldaspossibleinordertobeasflexibleandusefulasitcanbe.Forexample,thePostgreSQLcookbookshouldnotconcernitselfwithfirewallrules,thatis,therealmofafirewallcookbook.Instead,anadministratorwouldleverageagenericfirewallmodelandacookbookwithaspecificfirewallimplementationsuchastheUFWcookbooktoprovidethosefeatures.Inthiscase,ifyouweretolookattheUFWcookbook,youwouldseetheufw::databagrecipemakinguseofdatabagstomakethefirewallrulesasflexibleaspossible.

TipUFWstandsforuncomplicatedfirewall,apopulariptables-basedfirewallrulegenerationpackageforLinuxthatcomeswithmanymoderndistributionsandeasesthemanagementofafirewallconfiguration.

Inthiscase,ufw::databagexpectsthatthereisaspecificdatabagnamedfirewallandinsideofitareitemsthatsharenameswithrolesornodes;thisisinlinewiththenotionthatdatabagsarefree-form,butcertaincookbooksexpectcertainstructure.Ifourinfrastructuremodelhadtworoles,web_server,anddatabase_server,thenourfirewalldatabagwouldhavecontainedtwoitemsnamedaccordingly.Theweb_serveritemcouldlooklikethefollowinghash:

{

"id":"web_server",

"rules":[{

"HTTP":{

"dest_port":"80",

"protocol":"tcp"

},

"HTTPS":{

"dest_port":"443",

"protocol":"tcp"

}

}]

}

Here,idoftheitemmapstothenameoftheitem,whichisalsothenameoftherole,sothattheufw::databagrecipeknowswheretofetchthedataitneedstobuilditsinternalfirewallrules.Tocomputeitslistoffirewallrulestoapply,theufw::databagrecipeexaminesthelistofrolesthatthenodeisconfiguredwithandthenloadsthecorrespondingdatafromtheitemsinthefirewalldatabag.

Asyoucansee,databagsallowyoutostorecentralized,free-formconfigurationdatathatdoesnotnecessarilypertaintoaspecificnode,role,orrecipe.Byusingdatabags,cookbooksforconfiguringusers,databases,firewalls,orjustaboutanypieceofsoftwarethatneedsshareddatacanbenefitfromthedatastoredinadatabag.

Onemightwonderwhywehavedatabagswhenwealreadyhaveattributedata,andthatwouldbeagoodquestiontoask.Attributesrepresentthestateofanodeataparticularpointintime,meaningthattheyaretheresultofacompactionofattributedatathatisbeingsuppliedtoanodeatthetimetheclientisbeingexecuted.WhentheChefclientruns,theattributedataforallthecomponentscontributingtothenode’srunlistisevaluatedatthattime,flattenedaccordingtoaspecificprioritychain,andthenhandedtotheclient.Incontrast,databagscontainarbitrarydatathathasnoattachmenttoaspecificnode,role,orcookbook;itisfree-formdatathatcanbeusedfromanywhereforanypurpose.Onewouldnot,forexample,belikelytostoreuserconfigurationdatainacookbookoronaspecificnodebecausethatwouldn’tmakemuchsense;usersexistacrossnodes,roles,andevenenvironments.Thesamegoesforotherdatasuchasnetworktopologyinformation,credentials,andotherglobaldatathatwouldbesharedacrossafleet.

Large-scaleinfrastructureOneofthemanybenefitsofChefisthepowertoapplyrolestonodesatscale.Thismeansthatonceyoudefineasetofrolesandsomesupportingrecipes,youcanapplythemtoonehostjustaseasilyasanyother.TherearemanyorganizationsthatmanageverylargeinfrastructureusingChef,includingcompaniessuchasFacebook,Ancestry,andRiotGames.WithChef,configuringonehundredhostsisasstraightforwardasconfiguringonehost.Beingabletoachievescalabilitygoalswhileremainingcost-effectiveisacriticalpartofrunningatechnologybusiness.Tothisend,Chefprovidestoolstoautomatethecreation,provisioning,maintenanceandterminationofvirtualhostsusingtheprovidedtools,whichcanhelpachievebothscalabilityandconservationofresources.ThenextchapterdiscusseshowtouseCheftoextendyourinfrastructureintothecloud.

SummaryNowthatyou’velearnedthekeyterminologythatChefusesanddissectedanexampleinfrastructureabit,youcanseethefollowing:

Infrastructurecanbedecomposedintothevariousrolesthatresources(nodes)playwithinthatinfrastructureAcombinationofrecipesandconfigurationdataprovideuswithrolesthatdescribeapartofouroverallinfrastructureChefanalyzestherolesappliedtohardwareresources(nodes)andgeneratesarunlistthatisspecifictothenodethatthoserolesarebeingappliedtoArunlististhencombinedwiththecookbooks,recipes,templates,andconfigurationdatatobuildaspecificsetofscriptsthatareexecutedonthenodewhenthechef-clientisrunWecanapplythesemethodologiesofautomatedconfigurationtocloudserversandphysicalsystemsalike.

NowthatyouunderstandhowChefmodelsinteract,let’stakealookathowwecangetstartedwithcloudservicesusingChef.

Chapter3.IntegratingwiththeCloudBeingabletoconfigurenewhostsautomaticallymeansthatifyououtgrowyourexistingresources,youcaneasilybringupnewserverstoincreaseyourcapacitywithverylittleeffort.TheChefcommand-linetool,knife,providestheabilitytoprovisionnewhostswithcloudservicesautomaticallyfromthecommandlineifconfiguredcorrectly.ThischapterintroducesyoutousingChefwithtwopopularcloudplatformproviders:AmazonEC2andRackspaceCloud.Here,youwilllearntouseknifewithbothoftheseprovidersinordertoperformthefollowing:

ProvisionnewhostsaccordingtoyourhardwareneedsBootstraptheChefclientandregisterhostswiththeChefserviceListyourexistingcapacityTerminateunneededcapacity

Youwillseethatallofthiscanbedoneusingthecommand-linetoolsprovidedwithouteverhavingtologintotheprovider’swebinterface,andallofthiswithinafewminutesoftime.

LeveragingthecloudCloudcomputingproviderssuchasRackspaceCloudandAmazonEC2provideon-demandcomputingpoweratthepushofabutton,afeaturethathasbecomeimmenselypopularwithdevelopersandsystemsadministratorsalike.Oneofthemosttoutedbenefitsofcloudcomputingiscostsavings;however,theseon-demandinstancescanbecomeveryexpensiveiftheyareleftrunning.Oftenthecapacityoftimewillbeconfiguredinordertohandlelarge-scaleeventsandthenleftonlinebecauseofthetimerequiredtoreconfigurethesystemsiftheyareneededagain.Asunderutilizedcapacityendsupcostingmoneyratherthansavingit,beingabletoreduceorexpandthecapacityquicklyandeasilywillhelpyoumatchyourcomputingneedswhilesavingbothtimeandmoney.

Thissectionspecificallylooksattwoofthemorepopularcloudproviders:AmazonEC2andRackspaceCloud;however,thereareothers,andthetechniquesdescribedherewillbebroadlyapplicabletoanyothersupportedcloudprovider.

AmazonEC2AmazonEC2isaverypopularcloud-computingplatform,andknifehassupporttomanageEC2instancesfromthecommandlinethroughtheknife-ec2plugin.ThefollowingstepsdemonstratehowyoucanworkwithEC2:

1. InstalltheEC2knifeplugin.2. SetupyourSSHkeysforusewithEC2.3. ConfigureknifewithyourAWScredentials.4. FindthedesiredAMI.5. Provisionanewhostwithknife.6. Bootstrapthenewlycreatedhost.7. Configurethenewhostwitharole.

InstallingtheEC2knifepluginAsofChef0.10,theec2subcommandshavebeenmovedfrombeingbuiltinknifetoanexternalgem,knife-ec2.InordertouseEC2commands,youwillneedtoinstallthegem,whichcanbedoneviathefollowingcommand:

geminstallknife-ec2

ThiswillinstallallofthegemdependenciesthattheEC2pluginrequires.

TipSomeofthecloudproviderpluginshaveconflictingdependencies,soitmaybebesttoleverageagemmanagerinordertoisolatethem.Forexample,usingRVMorrbenv,youmightcreateoneRubygemenvironmentperprovidersothatyoucouldswitchbackandforthwithasimplecommandsuchasrvmgemsetusechef-ec2.

SettingupEC2authenticationInordertomanageyourEC2hosts,youwillneedyourEC2key-pairproperlyregisteredwithSSHandyourAWSaccesskeyssetinyourknifeconfigurationfile.

Todothefirst,makesureyouhaveyourEC2SSHkeysdownloadedandregisteredwithyourSSHagent.OnewaytodothisistoaddthefollowingtoyourSSHconfigurationfile,typically,$HOME/.ssh/config:

Host*.amazonaws.com

ForwardAgentyes

CheckHostIPno

StrictHostKeyCheckingno

UserKnownHostsFile=/dev/null

IdentityFile~/.ssh/ec2_keypair.pem

InordertoconfigureyourAWSkeys,youwillneedtoaddsomeinformationtoyourknife.rbconfigurationfile($HOME/.chef/knife.rb):

knife[:aws_access_key_id]="YOURACCESSKEY"

knife[:aws_secret_access_key]="SECRETKEY"

ThesekeystellknifewhichAWScredentialstousewhenmakingAPIcallstoperformactionssuchasprovisionnewhostsandterminateinstances.Withoutthis,knifewillbeunabletomakeAPIcallstoEC2.Withtheserequiredchangesmade,let’slookathowtocreateanewEC2instancewithknife.

ProvisioninganinstanceInitially,wewilllookatprovisioninganinstanceusingoneoftheUbuntuAMIs.Withknife,wecanspecifytheAMItouse,theavailabilityzonetotarget,andthesizeinstancetobecreated.Forexample,tocreateanm1.largesizeintheus-east-1eavailabilityzonewithUbuntu12.04.3LTS,wewouldneedtousetheAMIwithami-23447f4aasitsidentifier.

InordertodeterminetheAMIID,youwillneedtolookitupatthefollowingURL:

http://uec-images.ubuntu.com/

TipRememberthatwhendecidingwhichAMItouse,someoftheEC2instanceswillbe32bitandsome64bit;choosetheappropriateAMIbasedontheinstancetype,region,andstoragemethodyouwanttouse.

Theprogressofprovisioningcanbeseenusingthefollowingcommand:

$knifeec2servercreate-Iami-23447f4a-fm1.large-Zus-east-1e-N

<nodename>-xubuntu--sudo

Theoutputfromthepreviouscommandwillshowyoutheprogressoftheprovisioning(thismaytakeaminuteortwo,dependingontheregion,instancesize,howlongstatuscheckstake,andsoon):

[user]%knifeec2servercreate-Iami-23447f4a-fm1.large-Sec2-keypair

-Zus-east-1e–N<nodename>-xubuntu--sudo

InstanceID:i-0dfec92d

Flavor:m1.large

Image:ami-23447f4a

Region:us-east-1

AvailabilityZone:us-east-1e

SecurityGroups:default

Tags:Name:i-0dfec92d

SSHKey:ec2-keypair

Waitingforinstance…..............

PublicDNSName:ec2-54-80-59-97.compute-1.amazonaws.com

PublicIPAddress:54.80.59.97

PrivateDNSName:ip-10-157-31-234.ec2.internal

PrivateIPAddress:10.157.31.234

BootstrappingtheinstanceAsyoucansee,knifewilltellyouthepublicIPandpublicDNSnameofthenewinstancealongwiththeinstanceID,tags,andsoforth.Oncetheinstanceisprovisionedandisonline,itwillneedtobebootstrapped.Rememberthatbootstrappingwillinstallthe

ChefclientandregistertheinstancewiththeChefservice,whichwecandointhesamewaywebootstrapanyotherhost:

$knifebootstrap<instance-public-ip-address>-N<node-name>-xubuntu--

sudo

AsEC2provisionseachinstancewithanubuntuuserthathassudoprivileges,weprovidethebootstrapcommandwith–xubuntuand--sudotoensurewehavetherequiredprivilegestoperformthebootstrapping.Additionally,asyoumorethanlikelydonotwanttheAWS-providedDNSnameasthenodename,theChefnodenameissetthroughthe-N<node-name>commandlineflag.Oncethebootstrapstepisfinished,assumingthattherearenoerrors,verifythatyournewlyprovisionedhostislistedinyourchefservice:

$knifenodelist

TheoutputwillcontainyournewlybootstrappednodeID,asspecifiedbyyouinthecommandlineortheDNSname,ifyoudon’tspecifyanodename.YouhavenowprovisionedanewEC2instanceandregistereditwithyourChefservicewithonlytwocommands!

TerminatingtheinstanceOnceyouaredonewithtesting,youmaynotwanttoleavetheEC2instancerunning,asitwillincurcostsifitremainsidle.Toensurethisdoesn’thappen,performthefollowingfoursteps:

1. ListyourEC2instances2. DeletetheserverfromEC23. RemovetheserverfromChef4. VerifythattheinstancenolongerexistsinCheforEC2

TolistourEC2instances,usetheserverlistsubcommandoftheec2command,whichwilllistalloftheEC2instancesinthespecifiedregion.Ifyoudonotspecifyaregion,us-east-1isthedefaultregion.ThefullcommandtolistEC2serversisasfollows:

$knifeec2serverlist

Asanexample,executingthiscommandafterprovisioningthefirsthostwillshowatableofoneinstanceasfollows:

InstanceIDNamePublicIPPrivateIP

i-0dfec92di-0dfec92d54.80.59.9710.157.31.234

Formostknifecommands,youwillneedtheinstanceIDsotheprevioustablecanbetruncatedtofitinprint.

TipListingEC2nodeswillresultinatablethatcontainsallthecurrentlyprovisionedEC2instancesintheregionbymeansoftheEC2API,whichisseparatefromtheChefserviceAPI.ThismeansyouwillgetalistofalltheinstancesinEC2whetherornottheyare

registeredwithChef.ThefulltablewillcontainmostoftheinformationyoucanseeontheEC2controlpanel,includingthepublicandprivateIP,flavor,AMI,SSHkey,andsoon.

Deletinganinstanceisjustaseasyascreatingorlistingthem.Here,theserverdeletesubcommandisinvokedwiththeinstanceidentifiertobeterminated.ThiswillusetheEC2APItoissueaterminatecommand—thisisnotreversibleandsothecommandwillpromptyoutoensurethatyoureallydidwanttodeletetheinstance:

[user]%knifeec2serverdeletei-0dfec92d

InstanceID:i-0dfec92d

Flavor:m1.large

Image:ami-23447f4a

Region:us-east-1

AvailabilityZone:us-east-1e

SecurityGroups:default

SSHKey:ec2-keypair

RootDeviceType:instance-store

PublicDNSName:ec2-54-80-59-97.compute-1.amazonaws.com

PublicIPAddress:54.80.59.97

PrivateDNSName:ip-10-157-31-234.ec2.internal

PrivateIPAddress:10.157.31.234

Doyoureallywanttodeletethisserver?(Y/N)

WARNING:Deletedserveri-0dfec92d

WARNING:Correspondingnodeandclientforthei-0dfec92dserverwerenot

deletedandremainregisteredwiththeChefServer

RemovingtheChefnodeAtthispoint,theEC2instanceisbeingterminatedandremovedfromyouraccount.However,itisnotremovedfromtheChefservicethatneedstobedoneseparatelywiththenodedeletecommand.Here,theChefnodenameisspecified,nottheinstanceidentifier:

$knifenodedeletemy-first-ec2-instance

VerifythatthenodewasremovedfromChefusingnodelist:

$knifenodelist

TheoutputshouldshowyouthatyourEC2instanceisnolongerregisteredwithChef.

RackspaceCloudRackspaceCloudisanotherpopularcloud-computingproviderthatiswellsupportedbyChef.SimilartoEC2,thereisaknifepluginforRackspaceCloud:

geminstallknife-rackspace

InthesamewaythatAWSrequiresasetofcredentialstointeractwiththeAPItocreateandterminateinstances,RackspaceCloudhasitsownconfiguration.However,theRackspaceCloudAPIisalittlesimpler;youwillneedtoprovideknifewithyourRackspaceCloud’susernameandAPIkey.ForthosewhodonotalreadyhavetheirAPIkey,itcanbefoundinyourRackspaceCloudcontrolpanel.Theappropriateconfigurationtoaddtoyourknife.rbfileisasfollows:

knife[:rackspace_api_username]="YourRackspaceAPIusername"

knife[:rackspace_api_key]="YourRackspaceAPIKey"

Thisdatacanbehardcodedintoyourconfigurationfile,orsincetheknifeconfigurationfileisjustRuby,itcanbegeneratedbyevaluatingenvironmentvariablesorlookingatalocalfile.Thisisusefulifyouaresubmittingyourknife.rbfileintoasourcerepositorysothatcredentialsarenotleaked.

ProvisioninganinstanceRackspaceCloudserverprovisioningisjustasstraightforwardasitiswithEC2.Thereissomevariationinthecommand-lineoptionspassedtoknifebecauseofthewayRackspaceprovidesimagesforsystems.InsteadofusingtheinstancesizeandanAMI,youcanspecifytheflavorofthesystemtoprovision(thenode’sCPU,memory,anddiskallocation)andtheoperatingsystemtoimagetheinstancewith.Inordertodeterminewhatflavorsareavailable,theknife-rackspacepluginprovidestherackspaceflavorlistsubcommand:

$kniferackspaceflavorlist--rackspace-region=IAD

Asitispossiblethattherearedifferentcapacitiesindifferentregions,itisagoodideatocheckwhatisavailableintheregionwhereyouwanttoprovisionanode.Thiswillresultinalistofflavorsandtheirspecifications;asofnow,someofthecurrentofferingsinIADareasfollows:

IDNameVCPUsRAMDisk

2512MBStandardInstance151220GB

31GBStandardInstance1102440GB

42GBStandardInstance2204880GB

performance1-11GBPerformance1102420GB

performance1-22GBPerformance2204840GB

performance2-120120GBPerformance3212288040GB

performance2-1515GBPerformance41536040GB

Inadditiontoknowingwhichflavortoprovision,youneedanimageidentifier(similartoanAMI)toapplytothenewhost.Again,thislistmayvarywithregionandpossiblychangeovertimesothereisacommand,rackspaceimagelist,tolistthevarious

images:

$kniferackspaceimagelist--rackspace-region=IAD

Theoutputhereisquitelong,soithasbeensampledtoshowenoughtobeuseful:

IDName

ba293687-4af0-4ccb-99e5-097d83f72dfeArch2013.9

41e59c5f-530b-423c-86ec-13b23de49288CentOS6.5(PVHVM)

857d7d36-34f3-409f-8435-693e8797be8bDebian7(Wheezy)

896caae3-82f1-4b03-beaa-75fbdde27969Fedora18(SphericalCow)

fb624ffd-81c2-4217-8cd5-da32d32e85c4FreeBSD9.2

1705c794-5d7e-44d6-87da-596e3cf92144RedHatEnterpriseLinux6.5

df27d481-63a5-40ca-8920-3d132ed643d9Ubuntu13.10

d88188a5-1b02-4b37-8a91-7732e42348c1WindowsServer2008R2SP1

Asyoucansee,thereareanumberofLinux,BSD,andWindowsdistributionsavailabletoprovision.Inordertoprovisionanewhost,youwillusetheservercreatecommand,similartotheEC2command.Thefollowingknifecommandwillprovisiona512MBhostwithUbuntu13.10intheIADdatacenter:

$kniferackspaceservercreate-Idf27d481-63a5-40ca-8920-3d132ed643d9-f

2--rackspace-region=IAD

AssoonastheAPIrespondstotherequesttoprovisionanewhost,youwillseetheRackspacemetadataforthehost,suchastheinstanceID,name,flavor,andimage:

InstanceID:993d369f-b877-4f0f-be4b-cfc45c240654

Name:rs-21230044929009695

Flavor:512MBStandardInstance

Image:Ubuntu13.10(SaucySalamander)

Shortlyafterthis—oncethesystemhasbeenprovisioned,thenetworkinterfaceshavebeenconfigured,andtherootpasswordhasbeenset—theIPandrootpasswordwillbedisplayed:

PublicDNSName:162.209.104.248.rs-cloud.xip.io

PublicIPAddress:162.209.104.248

PrivateIPAddress:10.176.65.92

Password:yZ3D3Tck8uGm

AfterSSHbecomesavailable,knifewillinitiatetheprocessofbootstrappingthehost.Bydefault,knifewillusethechef-fulltemplate,whichwillinstallChefviatheomnibusinstallerfortheplatformyouarebootstrapping.Thiscanbealteredbyprovidingknifewiththe–dcommand-lineoption.Assumingthatthehostisbootstrappedproperly,thesystemdatawillbedisplayedonceagainforyourinformation:

InstanceID:993d369f-b877-4f0f-be4b-cfc45c240654

HostID:c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00

Name:rs-21230044929009695

Flavor:512MBStandardInstance

Image:Ubuntu13.10(SaucySalamander)

PublicDNSName:162.209.104.248.rs-cloud.xip.io

PublicIPAddress:162.209.104.248

PrivateIPAddress:10.176.65.92

Password:yZ3D3Tck8uGm

Environment:_default

Oncethebootstrapstepisfinished,assumingthattherearenoerrors,verifythatyournewlyprovisionedhostislistedinyourchefservice:

$knifenodelist

TheoutputwillcontainyournewlybootstrappednodeIDasspecifiedbyyouinthecommandline(via-N)orthenamegeneratedbyRackspace(inthisexample,itwillbers-21230044929009695).Congratulations!YouhaveprovisionedanewRackspaceinstancewithasinglecommand.

TerminatinganinstanceOnceyouaredonewithtesting,youmaynotwanttoleavetheEC2instancerunning,asitwillincurcostsifitremainsidle.Toensurethisdoesn’thappen,performthefollowingfoursteps:

1. ListyourRackspaceservers.2. DeletetheserverfromRackspace.3. RemovetheserverfromChef.4. VerifythattheinstancenolongerexistsinCheforRackspace.

TolistyourRackspaceinstances,usetheserverlistsubcommandoftherackspacecommand,whichwilllistalloftheRackspaceinstancesinthespecifiedregion.SimilartotheoutputfromtheEC2serverlistcommand,theoutputwilllooklikethefollowing:

$kniferackspaceserverlist--rackspace-region=IAD

InstanceIDName

993d369f-b877-4f0f-be4b-cfc45c240654rs-21230044929009695

TipSimilartotheEC2output,theresultingtableistoowideforprintsoonlytheinstanceIDandnodenameisshown.YoushouldexpecttoseepublicandprivateIPaddresses,instancetypes,andsomeotherdatathatyouwillbeabletoseeontheRackspaceCloudcontrolpanelaswell.

Youcandeleteaninstanceusingasinglecommand;theserverdeletesubcommandisinvokedwiththeRackspaceinstanceidentifiertobeterminated.Rememberthatthisisnotreversible,sothecommandwillpromptyoutoensurethatyoureallydowanttodeletetheinstance:

$kniferackspaceserverdelete993d369f-b877-4f0f-be4b-cfc45c240654--

rackspace-region=IAD

InstanceID:993d369f-b877-4f0f-be4b-cfc45c240654

HostID:c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00

Name:rs-21230044929009695

Flavor:512MBStandardInstance

Image:Ubuntu13.10(SaucySalamander)

PublicIPAddress:162.209.104.248

PrivateIPAddress:10.176.65.92

Doyoureallywanttodeletethisserver?(Y/N)y

WARNING:Deletedserver993d369f-b877-4f0f-be4b-cfc45c240654

WARNING:Correspondingnodeandclientforthe993d369f-b877-4f0f-be4b-

cfc45c240654serverwerenotdeletedandremainregisteredwiththeChef

Server

RemovingtheChefnodeAtthispoint,theEC2instanceisbeingterminatedandremovedfromyouraccount.However,itisnotremovedfromtheChefservice;thisneedstobedoneseparatelywiththenodedeletecommand.Here,theChefnodenameisspecified,nottheinstanceidentifier:

$knifenodedeleters-21230044929009695

VerifythatthenodewasremovedfromChefwithnodelist:

$knifenodelist

TheoutputshouldshowyouthatyourrecentlycreatedRackspaceinstanceisnolongerregisteredwithChef.

SummaryTheabilitytoscaleyourinfrastructurethroughacombinationofon-andoff-sitehostsisincrediblypowerful.Ifyouneedmorecapacity,youcaneasilybringupnewhostsonEC2,RackspaceCloud,oranysimilarplatform.Additionally,thesetechniquesapplytonotonlypubliccloudprovidersbutalsotoprivatecloudplatformssuchasVMWarevSphereandothers(provided,asuitablepluginforknifeexists).

Asyouhaveseen,withChefyoucanspinupandspindowntheservercapacitytomeetyourneedswithverylittleeffort.Onceyourinfrastructuremanagementisautomated,youcanfocusonhigherlevelproblemssuchasbuildingscalableservicesandscalingtomeetyourcustomers’demands.

Expandingonthiscapability,youcouldusethesetoolstoperformthefollowing:

ManuallyincreaseordecreasethecapacityinordertomatchthedemandWriteatooltoanalyzethecurrentresourceloadandreactaccordinglyPredictthefuturecapacityandscaleappropriatelyonagivenschedule

Nowthatwehavetheabilitytobringupsomehoststoworkwith,wecantakealookathowtoworkwithcookbookstolearnhowtheyworkandhowtobuildnewones.

Chapter4.WorkingwithCookbooksCookbooksareoneofthefundamentalcomponentsoftheChefsystem.Theyarecontainersforrecipes,providers,resources,templates,andallthelogicandinformationrequiredtomanageyourinfrastructure.Thischaptercoversthefollowing:

OrganizationofcookbooksBuildingcookbooksDevelopingrecipesHandlingmultipleplatformsforacookbookorganization

CookbooksareoneofthecorecomponentsofChef.Theyare,astheirnamesuggests,acollectionofrecipesandotherdatathatwhencombinedprovideaspecificsetoffunctionalitytoasystemadministrator.Ineachcookbook,youwillfindacollectionofdirectoriesandfilesthatdescribethecookbookanditsrecipesandfunctionality.Thecorecomponentsofacookbookareasfollows:

CookbookmetadataAttributesRecipesTemplatesDefinitionsResourcesProvidersRubylibrariesSupportfiles

Acookbookisacollectionoffilesanddirectorieswithawell-knownstructure.Noteverycookbookhasallofthesecomponents.Forexample,theremaybenoneedtodevelopcustomresourcesorprovidersinacookbookthatonlyusesChef-suppliedresources.However,everycookbookdoesneedtohaveametadatafilethatprovidesvariousbitsofinformationsuchasitsname,version,dependencies,andsupportedsystems.

Let’stakealookatthememcachedcookbookasitisareasonablysimplecookbookthatiscapableofinstallingandconfiguringthememcacheserver,whichisadistributedmemory-backedcacheservice:

Here,youcanseethatthiscookbookcontainsattributes,definitions,recipes,andtemplates,aswellasafilenamedmetadata.rb(themetadatafile)andaREADME.mdfile.Itisagoodideatoprovideexamplesofhowtouseyourcookbookandrecipesinsomesortofdocumentation,suchasaREADME.mdfile.Whenyoulookattheprecedingscreenshot,youwillseethatthedirectorynamesmaptothecomponentnamesandeachcontainssomefilesorsubdirectorieswithfiles.WewilldiscusstheorganizationofthespecificChefcomponentsingreaterdetailfurtheronaswediveintomoredetailsoneachtypelater.Fornow,itissufficienttoknowthatthedirectorystructureisdesignedtogrouptogetherfilesforeachtypeofcomponent.Alsonoticethat,asmentionedearlier,thiscookbookisanexampleofonethatdoesnothaveallthecomponents,astherearenonewresourcesorprovidersinthiscookbook.

SomeofthesefilesarepurelyinformationalandhavenoeffectonyourrecipesorChefitself,suchastheREADME.mdfile.Thisfile,andotherssuchasCHANGELOG,LICENSE,orDEVELOPMENTfiles,isincludedtoconveyinformationtoyouabouthowtoparticipate,license,orotherwiseusethecookbook.

Thereisalotofinformationthatisstoredinsideacookbook—thisinformationincludesthestepstotakeinordertoachieveadesiredeffectsuchastheinstallationofaserviceorprovisioningofusersonahost.Ahigh-leveloverviewofthecontentthatwewillbelearningaboutinthischapter,sothatyouhaveanideaofhowthecomponentsworktogetherbeforeyoulearnaboutthemindepth,isshownasfollows:

Attributes:Theseareattributesthatthecookbook’srecipesrelyon.Awell-definedcookbookshouldcontainsomesanedefaultsfortherecipessuchasinstallationdirectories,usernames,downloadableURLs,andversionnumbers.Anythingarecipeexpectsthenodetohavedefinedshouldbegivenadefaultvaluesothattherecipewillbehaveasexpected.Recipes:Rubyscriptsdefinetherecipesinthecookbook.Acookbookcancontainasfewasoneorasmanyrecipesasitsauthorwouldliketoputintoit.Mostpackage-specificcookbooksonlycontainafewrecipes,whilesomecookbooks,suchasyour

organization’sprivatecookbook,mayhavedozensofrecipesforinternaluse.Templates:TheseareRubyERBfilesthatareusedtodescribeanyfilethatneedstohavesomedynamicdatainit;often,theseareusedforstartupscriptsorconfigurationfiles.Resources:Thesedescribearesourcethatcanbeusedinarecipe.ResourcesareRubyscriptsthatuseChef’sresourcedomain-specificlanguage(DSL)todescribevariousactions,attributes,andotherpropertiesoftheresource.Providers:Thesedescribeanimplementationofaresource;inthecaseofthesupervisordcookbook,theserviceproviderfileoutlinestheactualimplementation-specificlogicoftheactionsthataresourcecanperform.Therearemanytypesofservicesthatyoucouldhave:supervisord,runit,monit,bluepill,andsoon.

Additionally,cookbooksmayincludeavarietyofsupportfilesthatarenotdirectlypartoftherecipes,suchasthefollowing:

Definitions:Theseareusedtobuildreusabletemplatesforresources.Perhapsyouwanttodefinethestructureofauseraccount,abackgroundworker,orarunnableprocess.Theseareawaytoprogrammaticallydescribewhattheselooklikeandimplementanylogictheymightneed.Rubylibraries:Anyreusablecodethatyourrecipesneedcanbeincludedinthecookbook.Thingsthatgoinhereareaccessiblebyyourrecipesandautomaticallyloadedforyou.SupportFiles:Thesearearbitrarydatafilesthatdon’tfallintoanyoftheothercategories.Tests:Recipes,composedofRubycode,canincludeunittestsorcucumberteststoverifythatthelogicworks.Notethatthesetestsareunittests,notintegrationtests;theyarenotdesignedtoensurethatyouhaveconfiguredyournodesproperlyorthattherearenoconflictsorotherissueswhenapplyingtheserecipes.

AttributesAttributesareChef’swayofstoringconfigurationdataandcanbethoughtofasalarge,butdisjointed,hashstructure.Chefpullsdatafromvariouslocationsandcombinesthatdatainaspecificordertoproducethefinalhashofattributes.Thisdataiscomputedwhenaclientrequestsitsrunlistfromtheserver(suchaswhenyouexecutechef-clientonanode).Thismechanismallowsyoutodescribedatawithahigherlevelofspecificityateachstepoftheprocess,decreasinginscopegoingfromthecookbookattributesfilesdowntonode-specificconfigurationdata.

Forexample,imagineyouaredeployingPostgreSQLontothehostsinyourinfrastructure.WithPostgreSQL,thereareaverylargenumberofconfigurationoptionsthatcanbetuned,rangingfromopenportsandnumberofconcurrentconnectionsdowntomemoryusedforkeycachesandotherfine-grainedconfigurationoptions.Thecookbook’sattributesfilesshouldprovideenoughconfigurationsforPostgreSQLtoworkwithoutmakinganymodificationtoahostandwithoutotherthingsbeingdeployed;also,theywouldmostlikelycontainaprettyvanillasetofconfigurationvalues,whichatahighlevelmightlooklikethefollowing:

InstallVersion9.3fromthesourcecodeListenonPort5432onIP0.0.0.0/0Storedatain/var/lib/postgresqlCreateanduseapgsqluser

Attributedatahasnotonlyanumberofsourcesthatitcanbepulledfrombutalsoasetofpriorities:default,normal,andoverride(inincreasingorder).Withineachlevel,dataispulledfromthecookbookattributesfilesandthenfromtheenvironment,role,andnodeconfigurationdatastoredintheChefserver(inthatorder).Combined,thisprovidesacomprehensivemechanismtodefineandcustomizethebehaviorofyourrecipesastheyareappliedtothenodes.

Now,asyoucanimagine,thisisfineforanumberofinstallationswheretheserverhasallofthespaceallocatedontherootmountpoint,ordoesn’thavesecurityrestrictionsaboutwhichIPaddressesshouldbelistenedon,andsoon.Itwouldbenicetobeabletosaythatinproduction,wewanttouseVersion9.3,butinatestenvironment,wewanttoinstallVersion9.4inordertoperformsometeststhatwedon’twanttoruninproduction.Wemayalsowanttospecifythatinproduction,ourhostsareEC2instanceswithacustomizedEBSRAIDforourPostgreSQLdataandsothedatashouldbestoredin/vol/ebs00/postgresql.Usingthismultilayeredapproachforconfigurationdata,thisisentirelypossible.

AttributefilescontainRubycodethatstorestheconfigurationdata.Inthiscase,toachieveourdescribeddefaultbehavior,wecouldhaveafile,attributes/default.rb,thatcontainsthefollowingtext:

default['postgresql']['port']="5432"

default['postgresql']['listen_address']="*"

default['postgresql']['data_dir']="/var/lib/postgresql"

default['postgresql']['install_method']="source"

default['postgresql']['version']="9.3"

ThehashthatthisdescribeswilllooklikethefollowingJSONdictionary:

'node':{

'postgresql':{

'port':'5432',

'listen_address':'*',

'data_dir':'/var/lib/postgresql',

'install_method':'source',

'version':'9.3'

}

}

Now,asdescribed,wewanttooverridetheversioninourstagingenvironmenttoinstallVersion9.4;thismeansthatinourstagingenvironment,theconfiguration(exactlyhowtomakethischangewillbediscussedlater)willneedtohavethefollowinginformation:

'node':{

'postgresql':{

'version':'9.4',

}

}

WhentheChefclientrunsonanodeinthestagingenvironment,theChefserverknowsthatthenodeisinthestagingenvironmentandwilltakethestageconfigurationaboveandoverlayitontopofthedefaultsspecifiedinthecookbook.Asaresult,thefinalconfigurationdictionarywilllooklikethefollowing:

'node':{

'postgresql':{

'port':'5432',

'listen_address':'*',

'data_dir':'/var/lib/postgresql',

'install_method':'source',

'version':'9.4'

}

}

Noticethattheversionhasbeenchanged,buteverythingelseremainsthesame.Inthisway,wecanbuildveryspecificconfigurationsforourhoststhatpullinformationfromavarietyofplaces.

ItisimportanttonotethatbecausetheseareinterpretedRubyscripts,theircontentscanrangefromsimpleattribute-settingstatementstocomplexlogicusedtodetermineanappropriatesetofdefaultattributes.However,it’sworthrememberingthatthemorecomplicatedyourconfigurationis,theharderitmaybetounderstand.

MultipleattributefilesChefloadsattributefilesinalphabeticalorderandcookbookstypicallycontainonlyoneattributefilenameddefault.rb.Insomecases,itmakessensetoseparatesomeoftheattributesintoseparatefiles,particularlywhentherearealotofthem.Asanexample,thecommunity-maintainedMySQLcookbookhastwoattributefiles:server.rbfortheserverattributesandclient.rbwithclient-specificattributes.Eachfilecontainsanywherebetween50and150linesofRubycode,soitmakessensetokeepthemseparateandfocused.

SupportingmultipleplatformsTherearetimeswhenasimpleattributesfiledoesn’tmakesense,sobeingabletodynamicallydefinethedefaultsisveryuseful.Consideramultiplatformcookbookthatneedstoknowwhichgrouptherootuserisin.Thenameofthegroupwillvaryaccordingtotheoperatingsystemoftheendhost.IfyouareprovisioningaFreeBSDorOpenBSDhost,thenthegroupfortherootwillbewheel,butonanUbuntumachine,thegroupisnamedadmin.TheattributesfilecanuseplainoldRubycodeoroptionalChef-providedconveniencemethodssuchasvalue_for_platform,whichisaglorifiedbutcompactswitchstatement:

default[:users]['root'][:group]=value_for_platform(

"openbsd"=>{"default"=>"wheel"},

"freebsd"=>{"default"=>"wheel"},

"ubuntu"=>{"default"=>"admin"},

"default"=>"root"

)

LoadingexternalattributesSometimesitisusefultoloadattributesfromanothercookbook;ifyourcookbookistightlycoupledtoanothercookbookorprovidessomeextendedfunctionality,itmaymakesensetousethem.Thiscanbeachievedintheattributesfilebyusingtheinclude_attributemethod(again,thisisaChef-specificconveniencemethod).

Let’sconsiderthatyouwanttoknowtheportthatApacheisconfiguredtouse.Youcouldusetheportkeyfromtheapacheconfigurationsection,butitisnotguaranteedthatitexists(itmaynothavebeendefinedortherecipethatcontainsitmaynothavebeenloadedyet).Toaddressthis,thefollowingcodewouldloadthesettingsfromattributes/default.rbinsideoftheapachecookbook:

include_attribute"apache"

default['mywebapp']['port']=node['apache']['port']

Ifyouneedtoloadanattributesfileotherthandefault.rb,sayclient.rb,insidethepostgresqlcookbook,youcanspecifyitinthefollowingmanner:

include_attribute"postgresql::client"

Makesurethatanycookbooksyourelyonarelistedasadependencyinyourcookbook’smetadata.Withoutthis,theChefserverwillhavenowayofknowingthatyourrecipesor

configurationdatadependonthatcookbook,andsoyourconfigurationmayfailasaresultofthis.

UsingattributesOnceyouhavedefinedyourattributes,theyareaccessibleinourrecipesusingthenodehash.Chefwillcomputetheattributesintheorderdiscussedandproduceonelargehash,whichyouwillhaveaccessto.

TipChefusesaspecialtypeofhash,calledaMash.Mashesarehasheswithwhatisknownasindifferentaccess—stringkeysandsymbolkeysaretreatedasthesame,sonode[:key]isthesameasnode["key"]).

IfwehadthePostgreSQLattributesanduserattributesasspecifiedpreviously,withoutanyoverrides,thentheresultingconfigurationwilllooklikethefollowing:

'node':{

'postgresql':{

'port':'5432',

'listen_address':'*',

'data_dir':'/var/lib/postgresql',

'install_method':'source',

'version':'9.3'

},

'users':{

'root':{'group'=>'wheel'},

}

}

Thisdatacouldthenbeaccessedanywhereinourrecipesortemplatesthroughvariablessuchasnode[:postgresql][port]ornode[:users][:root][:group].Rememberthatthefinalversionofthenode’sconfigurationdataisdeterminedatthetimetheclientmakestherequestforitsconfiguration.ThismeansthatChefgeneratesasnapshotofthecurrentstateofthesystem,collapsedaccordingtoitsrulesofprecedence,forthatnodeandpassesittothehosttoperformitsoperations.

MetadataEachcookbookcontainsametadata.rbfileintherootdirectoryofthecookbookthatcontainsinformationaboutthecookbookitself,suchaswhomaintainsit,thelicense,version,containedrecipes,supportedplatforms,andthecookbook’sdependencies.ThecontentsofthisscriptareusedtogenerateaJSONfilethatdescribesthecookbook,whichisusedbytheChefserverfordependencyresolution,searchingandimportingintorunlists.

Thisisarequiredfileforacookbook,andhereisanexamplemetadata.rbfilefromthePostgreSQLdatabaseserver,whichisslightlymodifiedtofitthefollowing:

name"postgresql"

maintainer"Opscode,Inc."

maintainer_email"cookbooks@opscode.com"

license"Apache2.0"

description"InstallsandconfiguresPostgreSQL"

long_descriptionIO.read(File.join(

File.dirname(__FILE__),'README.md'

))

version"3.3.4"

recipe"postgresql","Includespostgresql::client"

recipe"postgresql::ruby","InstallsRubybindings"

recipe"postgresql::client",

"Installsclientpackage(s)"

recipe"postgresql::server","Installsserverpackages"

recipe"postgresql::server_redhat",

"InstallsRedHatserverpackages"

recipe"postgresql::server_debian",

"InstallsDebianserverpackages"

%w{ubuntudebianfedorasuseamazon}.eachdo|os|

supportsos

end

%w{redhatcentosscientificoracle}.eachdo|el|

supportsel,">=6.0"

end

depends"apt"

depends"build-essential"

depends"openssl"

Becausethemetadata.rbfileisaRubyscript,itallowsyoutousearbitraryRubycodeinsideofit.Here,forexample,thelong_descriptionentryisgeneratedprogrammaticallybyreadinginthecontentsofthesuppliedREADME.mdfile:

long_descriptionIO.read(File.join(File.dirname(__FILE__),

'README.md'))

Here,thePostgreSQLcookbooksupportsmultipleplatforms,soinsteadofwritingeachplatformthatissupportedonalineofitsown,youcouldusealoopsimilartotheoneusedinthemetadata.rbfile:

%w{ubuntudebianfedorasuseamazon}.eachdo|os|

supportsos

end

Additionally,ifitonlysupportscertainplatformswithaminimumversion,youcouldwritesomethingsimilartothefollowing,whichdeclaressupportforRedHat-baseddistributionsgreaterthan(orequalto)Version6.0:

%w{redhatcentosscientificoracle}.eachdo|el|

supportsel,">=6.0"

end

Inthiscookbook,thedependenciesarelistedlinebylinebutcouldberepresentedsimilarlyifyouhavealargenumber:

depends"apt"

depends"build-essential"

depends"openssl"

Dependenciescouldalsoberewrittenasfollows:

%w{aptbuild-essentialopenssl}.eachdo|dep|

dependsdep

end

Obviously,inthiscase,youaren’tsavinganyroom;however,ifyouhadtenormoredependencies,itcouldmakeitmorecompact.

AslongasyourRubycodeproducessomethingthatisacompatibleargumentorconfiguration,youcanbeascleverasyouwant.Takeadvantageofyourabilitytodynamicallygenerateaconfiguration.

RecipesRecipesarethecorecomponentofgettingthingsdone.Theyarescripts,writteninRuby,thatprovidetheinstructionstobeexecutedonendhostswhentheChefclientisrun.Recipesareplacedintherecipesdirectoryinsideofacookbook,andeachrecipeisdesignedtoachieveaspecificpurpose,suchasprovisioningaccounts,installingandconfiguringadatabaseserver,andcustomsoftwaredeployments.

Recipescombineconfigurationdatawiththecurrentstateofthehosttoexecutecommandsthatwillcausethesystemtoenteranewstate.Forexample,aPostgreSQLdatabaseserverrecipewouldhavethegoalofinstallingandstartingaPostgreSQLserveronanyhostthatrunstherecipe.Let’slookatafewpossiblestartingstatesandtheexpectedbehavior:

AhostwithoutPostgreSQLinstalledwouldbeginatthestateofnothavingtheservice;then,itwillexecutethecommandsrequiredtoinstallandconfiguretheserviceHostswithanexistingbutoutdatedPostgreSQLservicewouldbeupgradedtothelatestversionofthedatabaseserverHostswithacurrentinstallationofPostgreSQLwouldhaveitsPostgreSQLinstallationuntouchedInallcases,theconfigurationonthediskwouldbeupdatedtomatchtheconfigurationstoredintheChefserver

Toachievethesegoals,recipesuseacombinationofresources,configurationdata,andRubycodetodeterminewhattoexecuteontheendhost.Eachhost-levelresource—files,configurationfiles,packages,users,andsoon—ismappedtoaChefresourceinarecipe.Forexample,considertherecipethatwesawearlierinthebookthatwasusedtodemonstratethattheChef-soloinstallationwasfunctional:

file"#{ENV['HOME']}/example.txt"do

action:create

content"Greetings#{ENV['USER']}!"

end

Thisisacompleterecipe;ithasonesteptocreateafile,andthatisallitdoes.Thefilebeingcreatedontheendhostneedsaname;here,itwillbenamedENV['HOME']/example.txt,whichisRuby’swayofrepresenting$HOME/example.txt.Inadditiontoaname,weareinstructingCheftocreatethefile(wecouldjustaseasilyinstructCheftodeletethefile)andtoputthecontentsGreetings$USERintothefile,replacingwhatisinthere.

Wecouldextendourrecipetoensurethataspecificuserexistedonthehostandcreateafilewiththeownersettothenewlycreateduser:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

end

file"/tmp/agent.txt"do

action:create

content"Hellofromsmith!"

owner"smith"

group"agents"

mode"0660"

end

Eachrecipeisascriptthatisrunfrombeginningtoend,assumingthatnothingcausesittoabort.Also,eachrecipecanaccessthenode’sattributedataandleverageresourcestocompiletemplates,createdirectories,installpackages,executecommands,downloadfiles,anddojustaboutanythingthatyoucandofromashellasanadministrator.Ifyoucan’taccomplishwhatyouneedtodousingtheexistingChefresources,youcaneithercreateyourowncustomresources,oryoucanexecuteanarbitraryuser-definedshellscript.

ResourcesResourcesareprogrammaticbuildingblocksinChef;theyareadeclarativemechanismtomanipulatearesourceonahost.Resourcesdeliberatelyhidetheunderlyingimplementationthatislefttoaprovider.Itisimportanttorecognizethataresourcedescribeswhatisbeingmanipulated,nothowitisbeingmanipulated;thisisbydesign,asitprovidesahighlevelofabstractionforChefrecipestobeasplatform-neutralaspossible.

Forexample,Chefhasbuilt-inresourcesthatincludethefollowing:

CronjobsDeploymentsFilesystemcomponents(mountpoints,filesanddirectories,andsoon)Sourcecoderepositories(Gitandsvn)LogsPowerShellscripts(Windowstargets)ShellscriptsTemplatesPackagesUsersandgroups

Resourcescombinedwithproviders(discussedshortly)arecollectivelyreferredtoasLWRPsorlightweightresourceproviders;theymakeupalargeportionofthefunctionalitywithinaChefrecipe.

Resourcesarecomposedofaresourcename(packagename,filepath,servicename,andsoon),anaction,andsomeattributesthatdescribethatresource.

UsingresourcesResources,aswehaveseeninsomeexamples,takethefollowingform:

resource_name<nameparameter><ruby_block>

Intheprecedingcode,resource_nameistheregisterednameoftheresource,suchasfile,package,anduser.Thenameparameterisaspecialargumenttotheresourcethatgivesthisresourceauniquename.Thisisoftenalsousedbytheresourceasadefaultvalueforanattributethatnaturallymapstothenameoftheresourcesuchasfilename,username,andpackagename(youcanseeapatternhere);however,youcanuseanarbitrarynameattributeandmanuallysettheattribute.TheRubyblockisjustablockofcodeinRuby;thisishowChef’sDSLworks.InChef,eachresourceexpectssomespecificthingsinsideitsconfigurationblock.Youwillfindthatmanyresourceshavedifferentexpectations,butingeneral,aresourceblockinarecipewillbeofthefollowingform:

resource_name"nameattribute"do

attribute"value"

attribute"value"

end

Thepreviousexample,whichcreatedanewuser,wasthefollowing:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

end

Here,theresourcenameisuser,thenameattributeissmith,andtheblockofcodebeingpassedtotheresourcehassevenattributes:action,system,comment,uid,gid,home,andshell.Eachoftheseattributeshasavalueassociatedwithit;internally,theRubycodefortheresourcewillstoretheseinsomevariablestobeusedwhenmanipulatingthespecifiedresource.Inthiscase,constructingauserontheendhostthroughthecorrectproviderwillbehelpful.

Oneoftheseattributes,action,isabitunique;itsjobistotelltheresourcewhatactiontotakeontheresource.Thelistofavailableactionswillbedifferentwitheachresource,buttypically,aresourcewillhaveactionssuchascreate,delete,orupdate.Havealookatthedocumentationfortheresourceyouareworkingwith;thedocumentationwilldescribetheavailableactionsandwhattheydoseparatelyfromtheotherattributes.

Todemonstratehowthenameattributeisusedasadefaultvaluefortheuserresource,thefollowingrecipehasthesamebehaviorasthepreviousone,buthasoneminorchange:

user"agent_smith"do

username"smith"

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

end

Here,youcanseethatanadditionalattribute,username,hasbeenaddedtotheresourceblockwiththevaluethatwaspreviouslyinthenameattribute.Additionally,thenameattributehasbeenchangedto"agent_smith".Ifyouweretoexecutethisrecipeorthepreviousexample,itwouldhavethesameeffect:tocreatealocalsystemuser,smith,withtheUID,GID,home,andotherattributesspecified.

OverridingadefaultbehaviorInadditiontoproperties,resourcesalsohaveadefaultaction.Moreoftenthannot,thedefaultactioniscreate,butagain,youwillwanttoconsultthedocumentationfortheresourceyouareworkingwithtomakesurethatyouknowwhatthedefaultbehaviorisforaresource.Youdon’twanttoaccidentallydestroysomethingyouthoughtyouwerecreating!

Aconcreteexamplemightbeinstallingthetcpdumppackageonyoursystem.Toinstallthedefaultversionwithnocustomization,youcouldusearesourcedescriptionsuchasthefollowing:

package"tcpdump"

Thisworksbecausethedefaultactionofthepackageresourceistoperformaninstallation.Ifyoulookatthesourcecodeforthepackageresource,youwillseethefollowingatthebeginningoftheconstructor:

definitialize(name,run_context=nil)

super

@action=:install

@allowed_actions.push(:install,:upgrade,

:remove,:purge,:reconfig)

@candidate_version=nil

@options=nil

@package_name=name

Thistellsusthatthedefaultaction,ifunspecified,istoinstallthepackageandtousethenameattributeasthepackagename.So,theprevioussimpleone-lineresourceisthesameaswritingoutthefollowingblock:

package"tcpdump"do

action:install

package_name"tcpdump"

end

Here,however,package_namewilldefaulttothenameattribute,sowedonotneedtoprovideitiftheresourcenameisthesameasthepackageyouwishtoinstall.Additionally,ifyouwantedtobemoreverbosewithyourresourcedescriptionandinstallaspecificversionofthetcpdumppackage,youcouldrewritethepackageresourcetolooksomethinglikethefollowing:

package"tcpdump"do

action:install

version"X.Y.Z"

end

Ifyoureadthedocumentationforthepackageresourceorexaminethefullconstructorforthepackageclass,youwillseethatithasanumberofotherattributesaswellaswhattheydoandwheretheirdefaultvaluescomefrom.Alltheresourcesfollowthisform;theyaresimplyRubyclassesthathaveanexpectedstructure,whichtheyinheritfromthebaseresourceclass.

TemplatesChefdynamicallyconfiguressoftwareandhosts,andalargepartofconfiguringUNIX-basedsystemsandsoftwareinvolvesconfigurationfiles.Chefprovidesastraightforwardmechanismtogenerateconfigurationfilesthatmakeiteasytocombineconfigurationdatawithtemplatefilestoproducethefinalconfigurationfilesonhosts.ThesetemplatesarestoredinthetemplatesdirectoryinsideofacookbookandusetheERBtemplatelanguage,whichisapopularandeasy-to-useRuby-basedtemplatelanguage.

Whyusetemplates?Withouttemplates,yourmechanismtogenerateconfigurationfileswouldprobablylooksomethinglikethis:

File.open(local_filename,'w')do|f|

f.write("<VirtualHost*:#{node['app]['port']}")

...

f.write("</VirtualHost>")

end

Thisshouldbeavoidedforanumberofreasons.First,writingconfigurationdatathiswaywouldmostlikelymakeyourrecipeveryclutteredandlengthy.Secondly,andmoreimportantly,itviolatesChef’sdeclarativenature.Bydesign,Chefprovidesyouwiththetoolstodescribewhattherecipeisdoingandnothowitisdoingit,whichmakesreadingandwritingrecipesmucheasier.Simplerrecipesmakeforsimplerconfiguration,andsimplerconfigurationscalesbetterbecauseitiseasiertocomprehend.Handrollingaconfigurationfileistheoppositeapproach;itveryspecificallydictateshowtogeneratethefiledata.Considerthefollowingcode:

config_file="#{node['postgresql']['dir']}/postgresql.conf"

pgconfig=node[:postgresql]

File.open(config_file'w')do|f|

f.write("port=#{pgconfig[:port]}")

f.write("data_dir=#{pgconfig[:data_dir]}")

f.write("listen_address=#{pgconfig[:listen_address]}")

end

File.chown(100,100,config_file)

File.chmod(0600,config_file)

ThiscodegeneratesaPostgreSQLconfigurationfilefromtheattributehash,onelineatatime.Thisisnotonlytime-consumingandhardtoreadbutpotentiallyveryerror-prone.Youcanimagine,evenifyouhavenotpreviouslyconfiguredanyPostgreSQLservers,justhowmanyf.write(...)statementscouldbeinvolvedingeneratingafullpostgresql.conffilebyhand.Contrastthatwiththefollowingblockthatleveragesthebuilt-intemplateresource:

template"#{node['postgresql']['dir']}/postgresql.conf"do

source"postgresql.conf.erb"

owner"postgres"

group"postgres"

mode0600

end

Theprecedingblockcouldbecombinedwithatemplatefilethatcontainsthefollowingcontent:

<%node['postgresql']sort.eachdo|key,value|%>

<%nextifvalue.nil?-%>

<%=key%>=<%=

casevalue

whenString

"'#{value}'"

whenTrueClass

'on'

whenFalseClass

'off'

else

value

end

%>

<%end%>

Ifwetakeourtemplateandthenapplythefollowingattributedataaswehadshownpreviously,thenwewouldhavegeneratedtheexactsameconfigurationfile:

'node':{

'postgresql':{

'port':'5432',

'listen_address':'*',

'data_dir':'/var/lib/postgresql',

'install_method':'source',

'version':'9.3'

},

'users':{

'root':{'group'=>'wheel'},

}

}

Onlynowwecanuseatemplatethatishighlyflexible.Ourtemplateusesthekey-valuecombinationsstoredintheconfigurationhashtodynamicallygeneratethepostgresql.conffilewithoutbeingchangedeverytimeanewconfigurationoptionisaddedtoPostgreSQL.

ChefusesERB,atemplatelanguage,thatisprovidedbythecoreRubylibrary.ERBiswidelyavailableandrequiresnoextradependencies;itsupportsRubyinsideoftemplatesaswellassomeERB-specifictemplatemarkup.

AquickERBprimerAsERBisverywelldocumentedandwidelyused,thisportionofthechapterservesonlyasaquickreferencetosomeofthemostcommonlyusedERBmechanisms.Formoreinformation,seetheofficialRubydocumentationathttp://ruby-doc.org/stdlib-2.1.1/libdoc/erb/rdoc/ERB.html.

ExecutingRuby

ToexecutesomearbitraryRubycode,youcanusethe<%%>container.The<%partindicatesthebeginningoftheRubycode,and%>indicatestheendoftheblock.Theblockcanspanmultiplelinesorjustonesingleline.Examplesofthisareasfollows:

ERBcode Output

<%

[1,2,3].eachdo|index|

putsindex

end

1

2

%> 3

<%users.collect{|u|

putsu.full_name}%>

BobSmith

SallyFlamingo

YoucanmixRubyandnon-Rubycode(usefultorepeatblocksofnon-Rubytext)asfollows:

<%[1,2,3].eachdo|value|%>

Non-rubytext…

<%end%>

Thiswouldyieldthefollowing:

Non-rubytext…

Non-rubytext…

Non-rubytext…

Variablereplacement

ERBhasasyntaxtoreplaceasectionofthetemplatewiththeresultsofsomeRubycoderatherthanrelyingonprintstatementsinsideyourRuby.Thatcontainerissimilartothelastone,withtheadditionoftheequalsigninsidetheopeningtag.Itlookslike<%=%>.AnyvalidRubycodeisacceptableinsidethisblock,andtheresultofthiscodeisputintothetemplateinplaceoftheblock.Examplesofthisareasfollows:

<%=@somevariable%>

<%=hash[:key]+otherhash[:other_key]%>

<%=array.join(",")%>

Thiscanbecombinedwiththepreviousexampletoproducecomplexoutput:

<%[1,2,3].eachdo|value|%>

Thevaluecurrentlyis<%=value%>

<%end%>

Thiswouldyieldthefollowing:

Thevaluecurrentlyis1

Thevaluecurrentlyis2

Thevaluecurrentlyis3

UsingjustthesebasicfeaturesofERB,combinedwithChef’sconfigurationdata,youcanmodeljustaboutanyconfigurationfileyoucanimagine.

ThetemplateresourceChefprovidesatemplateresourcetogeneratefilesviatemplates.Therearethreekeyattributesofthetemplateresource,whichareasfollows:

path:Thisspecifieswheretoputthegeneratedfilesource:Thistellstheresourcewhichtemplatefiletousevariables:Thisspecifieswhatdatatopopulatethetemplatewith

Thepathattributeusesthenameattributeasitsdefaultvalueandpopulatesthetemplatespecifiedbyasourcefilewiththedatapassedtoitthroughthevariablesattribute.Templatesarecontainedinsideofthetemplatesdirectory,whichisplacedinsideofacookbook;ifasourceisnotspecified,itwillbeexpectedthatafileexistsinsidethedirectorywiththesamenameasthepath,whichisonlyrootedinthetemplatesdirectorywitha.erbextension.Hereisasimpletemplateresourceexample:

template"/etc/motd"do

variables:users=>["Bart","Homer","Marge"]

end

Thisresourcewillexpectthatafileexistsinthetemplate’ssearchpath(moreonhowthatisdeterminedinabit)asetc/motd.erb,anditthenexposesanarrayofthreestringsasausersvariableandwritestheresultsoutas/etc/motdonthehost.ThecorrespondingMOTDtemplatecouldlooklikethefollowing:

Welcometocrabapple.mydomain.com!Ournewestusersare:

<%@users.eachdo|user|%>

*<%=user%>

<%end%>

ThetemplatevariablesTherearetwoprimarysourcesofdataforatemplate:datapassedexplicitlythroughtheresourceblockattributesandnodeconfigurationdata.Explicitvariablesareuserdefinedintherecipeandmaybeusedtooverridesomesettingsorpassinconfigurationthatisdynamicallygeneratedinsidetherecipe.ThenodeconfigurationdataiscomputedbyChefatruntimeandrepresentsasnapshotofthecurrentconfigurationthatwillbeappliedtothenode.

PassingvariablestoatemplateSometimesyouwillneedtopassdatatoatemplatefrominsideyourrecipeinsteadofrelyingontheglobalnodeattributes.Perhapsyouhavesomelogicthatcomputessomevariabledata,butitdoesn’tbelonginthenodehash;Chefsupportsdoingjustthisinthetemplateresource.ThedatapassedexplicitlyisavailabletotheERBtemplateasaninstancevariable,prefixedinRubywiththe@symbol.Forexample,considerthefollowingrecipesnippet:

config_hash={

:food=>"asparagus",

:color=>"blue"

}

template"/etc/myapp.conf"do

source"myapp.conf.erb"

owner"root"

mode"0664"

variables(

:install_path=>"/opt/#{hostname}/myapp",

:config=>config_hash

)

end

The:install_pathand:configkeysareaccessibleinthetemplateasinstancevariableswiththesamename.Theywillbeprefixedbythe@characterandcouldbeusedinatemplatesimilartothefollowing:

database_path="<%=@install_path%>/db"

storage_path="<%=@install_path%>/storage"

<%config.eachdo|key,value|%>

<%=key%>="<%=value%>"

<%end%>

Here,thetemplateexpectsaspecifickey,install_path,todeterminewheretostorethedatabase;thekey-valuehashspecifiedbyconfigisthenusedtogeneratesomearbitraryconfigurationsettingsinthetemplate.

AccessingcomputedconfigurationsInadditiontodatapassedviathevariablesattribute,atemplatecanaccessanode’scomputedconfigurationdatathroughthenodelocalvariable.ThisisaccessedasaRuby

hash,whichwillbestructuredsimilarlytoadictionaryorahashinanyotherlanguage.InourpreviousPostgreSQLattribute’sdataexample,thefollowingvaluesweredefined:

default['postgresql']['port']="5432"

default['postgresql']['listen_address']="*"

Evenifnootherconfigurationdatasupersedestheseconfigurationvalues,therewillbeapostgresqlkeyinthenode’sconfigurationdatathatcontainsthekey’sportandlisten_address.Usingthisinformation,wecanwritearecipethatusesatemplateresourceandamatchingtemplatelikethefollowing:

template"/etc/postgresql/postgresql.conf"do

source"postgresql.conf.erb"

owner"psql"

mode"0600"

end

listen_addresses='<%=node[:postgresql][:listen_address]'

port=<%=node[:postgresql][:port]%>

Whenthedefaultattributesdataiscombinedwiththeexampletemplate,theresulting/etc/postgresql/postgresql.conffilewillhavethefollowingcontent:

listen_addresses='*'

post=5432

Aspreviouslydiscussed,thecomputedattributeshashforagivennodecomesfromavarietyofsources.Thosesourcesincludeattributesfilesinthecookbook,role,environment,andnode-levelconfigurationvaluesstoredinChef,eachwithitsownlevelofprecedence.

SearchingfortemplatesAsyouhavelikelynoticed,Chefattemptstoallowyouasmuch,oraslittle,specificityasyouwantwhendefiningyourconfiguration,andtemplatesarenodifferent.Justasthefinalnodeconfigurationiscomputedfromavarietyoflocations,thetemplatesdirectoryhasaspecificsearchorder.Thisallowstheauthorofthecookbooktoprovideasetofdefaulttemplatesaswellassupportplatformandhost-specificoverrides.

Thedefaulttemplatedirectoryshouldbeusedtoprovidedefaultversionsofthetemplates.Anyplatform-orhost-specificdirectoriesareplacedalongsideitandwillbeusedwhenappropriate.Thesearchorderforatemplateisasfollows:

HostnameDistributionversionDistributionDefaultlocation

Asanexample,let’sconsiderascenarioinwhichweappliedarecipewiththepostgresql.conf.erbtemplateresourcetoanode,db1.production.mycorp.com,whichisrunningDebian6.0.Chefwillthenlookforthefollowingfilesinsideofthetemplatesdirectory:

host-db1.production.mycorp.com/postgresql.conf.erb

debian-6.0/postgresql.conf.erb

debian/postgresql.conf.erb

default/postgresql.conf.erb

Thesearchisperformedinthatorderwiththefirstmatchbeingthechosentemplate,applyingthehighestlevelofspecificitybeforethelowest(asisthepatternwithotherChefmechanisms,includingconfigurationdata).

Thisdifferentiationofconfigurationfilesbyhost,platform,andevenversionisveryuseful.Itallowsyoutoprovideasanesetofdefaultswhilesupportinghost-orsystem-specificquirkssimultaneously.

DefinitionsSometimes,youfindthatyouarecreatingsomethingrepeatedlyand,similartoaconfigurationtemplate,youneedatemplatetogenerateobjectsofagiventype.SomeexamplesofthismightbeApachevirtualhosts,aspecifictypeofapplication,oranythingelsethatisrepeatedalot.Thisiswheredefinitionscomein,andtheyarestoredinthedefinitionsdirectoryinsideofacookbook.

Definitionsareloadedandavailableasnamedresourcesjustasotherresourcessuchaspackages,files,andsoonare;theonlydifferenceisthatthereisnoprovider.Youcanthinkofthemasresourcesandprovidersallinone.Subsequently,theyaremuchmorerigidandlimitedinscopethananormalresourcewouldbe.HereisanexampledefinitiontoinstallPythonlibrariesusingpipandarequirements.txtfile:

define:pip_requirements,:action=>:rundo

name=params[:name]

requirements_file=params[:requirements_file]

pip=params[:pip]

user=params[:user]

group=params[:group]

ifparams[:action]==:run

script"pip_install_#{name}"do

interpreter"bash"

user"#{user}"

group"#{group}"

code<<-EOH

#{pip}install-r#{requirements_file}

EOH

only_if{File.exists?("#{requirements_file}")andFile.exists?("#

{pip}")}

end

end

end

Here,wearedeclaringanewtypeofdefinition,apip_requirementsobject.Thislooksandbehavessimilarlytoaresource,exceptthatitismuchsimpler(andlessflexible)thanaresource.Ithassomeattributes,whichareloadedviathespecialparamsargument,andcontainsalittlebitoflogicwrappedaroundascriptresource.Let’stakealookathowitwouldbeusedandthenseehowitworks:

pip_requirements"my_requirements"do

pip"#{virtualenv}/bin/pip"

usernode[:app][:user]

groupnode[:app][:group]

requirements_file"#{node[:app][:src_dir]}/requirements.txt"

end

Hereyouseewhatlookslikearesource,butisinfactadefinition.Asmentionedearlier,theselookverysimilarbecausetheybehavealike.However,youmusthavelikelynoticedthatthedefinitionofpip_requirementsitselfdidnothaveanysortofabstraction;thereisnopluggableprovider,novalidation,itdoesn’tsubclasstheResourceclass,amongother

differences.Definitionsprovideyouwithamechanismtodeclarereusablechunksofcodethatyourrecipeswouldotherwiseduplicatesothatyourrecipecanagaindescribethewhat,notthehow.

Thepreviousexampletellsusthatwehaveapip_requirementsobjectandthatwewanttopasssomeparameterstoit,namely,thepathtopip,theuserandgrouptorunpipas,andtherequirements.txtfiletoload.Thesearebroughtintothedefinitionthroughtheparamsargumentandcanbeaccessedasanyothervariabledata.Inthiscase,thedefinitionsaystorunbashasthespecifieduserandgroupandthatthescriptshouldruntheequivalentofthefollowing:

pipinstall-r/path/to/requirements.txt

Thiswillhappenonlyifpipandthe/path/to/requirements.txtfileexist(asindicatedbytheonly_ifguard).Bycreatingsuchadefinition,itcanbereusedanytimeyouneedtoinstallPythonmodulesfromaspecificrequirements.txtfileonyourhost.

RecipesRecipesarewhereallthemagichappenswithChef;theyarethesecretsauce,themanbehindthemask.TheyaretheworkhorsesofconfiguringhostswithChef.RecipesarescriptswritteninRubyusingChef’sDSLthatcontaintheinstructionstobeexecutedonendhostswhentheChefclientisrun.Everytimetheclientisexecutedontheendhost,afewthingshappen:

1. TheendhostmakesarequesttotheChefserversaying,“Ineedtodosomework”.2. TheChefserverlooksattherequestinghost’sidentityanddetermines:

Whichrecipesneedtoberunandinwhatorder(therunlist)Thecomputedconfigurationdataforthathost

3. Thisinformationispassedbacktotheendhostalongwiththenecessaryartifactsitneeds(recipes,templates,andsoon).

4. Theclientthencombinestheconfigurationdatawiththerecipesandbeginstoexecuteitsrunlist.

DevelopingrecipesAsadeveloper,youwillbeplacingyourrecipesinsidetherecipesdirectoryofyourcookbook.Eachrecipeisdesignedtoperformaspecificactionorsetofactionstoachieveagoalsuchasprovisioningaccounts,installingandconfiguringadatabaseserver,customsoftwaredeployments,orjustaboutanyotheractionthatyoucouldperformonaserver.

Akeyconceptwhendevelopingrecipesisthattheyshouldbeidempotent.Forthoseunfamiliarwiththeterm,anidempotentoperationisanoperationthatcanbeappliedmultipletimesandhavethesameoutcome.Considerthefollowingrecipe:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

end

Onewouldexpect,fromlookingatthisrecipe,thatChefshouldbeabletoexecuteitonce,fivetimes,oronethousandtimes,anditwouldhavethesameeffectastheinitialapplicationoftherecipe.Therewouldnotbefiveoronethousandusersonthehostwiththeloginnamesmith;therewouldbeonlyonesingleuserwiththeloginnamesmith.Also,inalltheruns,itwouldbeconstructedwiththesameUID1000,thesamegroupname,andsoon.

Similarly,givenaparticularstateofthesystemandassumingnothinghaschangedinbetweenruns,subsequentclientexecutionsshouldproducethesame,consistentendingstate.Inshort,theChefclientshouldbeabletoruntwotimesinarow,andiftheconfigurationhasnotbeenupdated,thesystemshouldlookexactlythesameafterthesecondrunasafterthefirstrun.

Recipesuseprovidedconfigurationdataalongwiththecurrentstateofthehosttodeterminetheflowandactionstakenbythescript.Theexecutionofarecipewilltakethesystemfromitsinitialstate,Sintial,toitsnewstate,Sfinal.Well-writtenrecipesshouldbeidempotentsuchthatifthey’reexecutedimmediatelyafterwardsanynumberoftimeswithnoconfigurationorstatchanges,thenthesystemshouldgofromSfinaltothesameSfinalwithnonewchangestothesystem.Thisallowsyoutokeepyoursystemsinaconsistentstateatalltimes,assumingthatnothinggoeswrongduringtheexecutionofthoseoperations;ifsomethingdoesgowrong,youshouldbeabletoreverttoapreviouslyknowngoodstate.

WritingrecipesAsyouhavealreadyseen,cookbooksprovideawaytocombinerelevantpiecesofconfigurationdatasuchasattributes,templates,resources,providers,anddefinitionsinoneplace.Theonlyreasonthesecomponentsexististosupportourrecipes.Recipescombineresourcesinacertainordertoproducethedesiredoutcome;muchinthesamewayachefwouldcombineingredientsaccordingtohisorherrecipetoproducesomedeliciousfood.Byputtingalloftheseresourcestogether,wecanbuildourownrecipesthatrangefromverysimplesingle-steprecipestomultistep,multiplatformrecipes.

StartingoutsmallAverybasicrecipe,aswehavediscussedbefore,mightonlyleverageoneortworesources.OneofthesimplestconceivablerecipesistheoneweusedearliertoverifythatourChef-soloinstallationwasworkingproperly:

file"#{ENV['HOME']}/example.txt"do

action:create

content"Greetings#{ENV['USER']}!"

end

Hereagain,wearecombiningasingleresource,thefileresource,specifyingthatwewanttocreatethefilenamed$HOME/example.txt,andstorethestring"Greetings$USER"inthatfile.$USERand$HOMEwillbereplacedbytheenvironmentvariables,mostlikelytheloginnameoftheuserthatisexecutingchef-clientandtheirhomedirectoryrespectively(unlesstheenvironmentvariableshavebeentamperedwith).

Followingourgoalofidempotence,executingthisrecipemultipletimesinarowwillhavethesameeffectasonlyrunningitonce.

InstallingasimpleserviceNowthatwe’vecoveredasimplerecipe,let’stakealookatonethatconfigurestheRedisengineandusessupervisordtorunthedaemon.Thisrecipedoesn’tinstallRedis;instead,itdefineshowtoconfigurethesystemtostartandmanagetheservice.Itdoesnothaveanyadvancedlogic,butmerelyconstructssomerequireddirectories,buildsaconfigurationfilefromatemplate,andthenusesthesupervisor_serviceresourcetoconfigurethedaemontorunandbemonitored,asshowninthefollowingcode:

redis_user=node[:redis][:user]

redis_group=node[:redis][:group]

environment_hash={"home"=>"#{node[:redis][:home]}"}

#Createthelogdiranddatadir

[node[:redis][:datadir],node[:redis][:log_dir]].eachdo|dir|

directorydirdo

ownerredis_user

groupredis_group

mode"0750"

recursivetrue

end

end

#Generatethetemplatefromredis.conf.erb

template"#{node[:redis][:config_path]}"do

source"redis.conf.erb"

ownerredis_user

groupredis_group

variables({:data_dir=>"#{node[:redis][:data_dir]}"})

mode0644

end

#Conveniencevariablesforreadability

stdout_log="#{node[:redis][:log_dir]}/redis-stdout.log"

stderr_log="#{node[:redis][:log_dir]}/redis-stderr.log"

redis_bin="#{node[:redis][:install_path]}/bin/redis-server"

redis_conf="#{node[:redis][:config_path]}"

#Tellsupervisortoenablethisservice,autostartit,runitas

#theredisuser,andtoinvoke:

#/path/to/redis-server/path/to/redis.conf

supervisor_service"redis_service"do

action:enable

autostarttrue

user"#{redis_user}"

command"#{redis_bin}#{redis_conf}"

stdout_logfile"#{stdout_log}"

stderr_logfile"#{stderr_log}"

directory"#{node[:redis][:install_path]}"

environmentenvironment_hash

end

Youwillnoticethatinordertokeeptheconfigurationconsistent,wereusealotofattributes.Forexample,thebeginningoftherecipeusesnode[:redis][:datadir]and

node[:redis][:log_dir]toensurethatthedirectoriesexistbymakinguseofadirectoryresourceinsideofaloop;then,theseareusedlaterontodefinethesupervisorconfigurationvariables(wheretowritelogs)andthetemplatefortheconfigfile(wheretostorethedata).Inall,thisrecipeiscomposedoffourresources:twodirectoriesintheloop,onetemplate,andonesupervisorservice.Bytheendofthisrun,itwillhaveensuredthecriticaldirectoriesexist,Redisisconfigured,andasupervisorconfigurationfileisgenerated(aswellaspokedsupervisordtoreloadthenewconfigurationandstarttheservice).Again,runningthismultipletimes,assumingnoconfigurationchangesinbetweenrunswillputthesystemintheexactsamestate.Rediswillbeconfiguredaccordingtothehostproperties,andsupervisorwillruntheservice.

GettingmoreadvancedLet’smoveupandtakealookataslightlymorecomplicated,yetfairlysimple,recipefromthegitcookbookthatinstallsthegitclientonthehost.Thecookbookismultiplatform,solet’stalkaboutwhatitwillbedoingbeforeitshowsyouthesource.Thisrecipewillbeperformingthefollowingactions:

1. Determinewhichplatformtheendhostisrunningon(byinspectingthenode[:platform]attribute).

2. IfthehostisrunningaDebian-baseddistribution,itwillusethepackageresourcetoinstallgit-core.

3. IfthehostisaRHELdistribution,itwillperformthefollowing:

1. IncludetheEPELrepositorybypullingintheepelrecipefromtheyumcookbookiftheplatformversionis5.

2. Usethepackageresourcetoinstallgit(asthatistheRHELpackagename).

4. IfthehostisWindows,itwillinstallgitviathewindows_packageresourceandinstructittodownloadthefilelocatedatnode[:git][:url](whichinturnpullsfromthedefaultattributesoroverriddenconfiguration),validatethatthechecksummatchestheonespecifiedbynode[:git][:checksum],andtheninstallit;however,thisisonlyiftheEXEisnotalreadyinstalled.

5. IfthehostisrunningOSX,itwillleveragethedmg_packageresourcetoinstalla.pkgfilefroma.dmgimage.Here,thedownloadURL,volumename,packagefile,checksum,andappnameareallattributesthatneedtobeprovided.

6. Finally,ifnoneoftheconditionsaremet,itfallsbacktothepackageresourcetoinstallthegitpackageinthehopethatitwillwork.

Hereisthecodeforthisrecipe:

casenode[:platform]

when"debian","ubuntu"

package"git-core"

when"centos","redhat","scientific","fedora"

casenode[:platform_version].to_i

when5

include_recipe"yum::epel"

end

package"git"

when"windows"

windows_package"git"do

sourcenode[:git][:url]

checksumnode[:git][:checksum]

action:install

not_if{File.exists?'C:\ProgramFiles(x86)\Git\bin\git.exe'}

end

when"mac_os_x"

dmg_package"GitOSX-Installer"do

appnode[:git][:osx_dmg][:app_name]

package_idnode[:git][:osx_dmg][:package_id]

volumes_dirnode[:git][:osx_dmg][:volumes_dir]

sourcenode[:git][:osx_dmg][:url]

checksumnode[:git][:osx_dmg][:checksum]

type"pkg"

action:install

end

else

package"git"

end

Onethingwehaven’tseenyetistheuseofthenot_ifqualifier.Thisisexactlywhatitlookslike;iftheblocksuppliedtonot_ifreturnsatruevalue,theresourcewillnotbeprocessed.Thisisveryusefultoensurethatyoudon’tclobberimportantfilesorrepeatexpensiveoperationssuchasrecompilingasoftwarepackage.

SummaryThischapterintroducedyoutothecriticalcomponentsofacookbookthatareusedtowriterecipes.Italsoshowedyousomeexamplerecipestogetyoustarted;thereareanumberofadvancedactionsthatcanbeaccomplishedinyourrecipes,suchassearchingtheChefserverfordata,loadingdatafromdatabags,orusingencrypteddata.Additionally,youcanaddmorecomponentstoyourcookbookssuchascustomresourcesandproviders,tests,andarbitraryRubylibraries.Allofthesewillbediscussedindetailinlaterchapters,butfirstlet’stakealookatwritingsomecompletecookbooks.We’llthenlearnhowtotestthembeforewemoveontolookingatsomecookbooksforcommonsystemadministrationtasks,andthenwe’llprogressontoadvancedtopics.

Chapter5.TestingYourRecipesSofar,youhaveseenhowtomodelyourinfrastructure,provisionhostsinthecloud,andwhatgoesintoacookbook.Oneimportantaspectofdevelopingcookbooksiswritingtestssothatyourrecipesdonotdegradeovertimeorhavebugsintroducedintotheminthefuture.Thischapterintroducesyoutothefollowingconcepts:

UnderstandingtestmethodologiesHowRSpecstructuresyourtestsUsingChefSpectotestrecipesRunningyourtestsWritingteststhatcovermultipleplatforms

Thesetechniqueswillprovetobeveryusefultowriterobust,maintainablecookbooksthatyoucanusetoconfidentlymanageyourinfrastructure.Testsenableyoutoperformthefollowing:

IdentifymistakesinyourrecipelogicTestyourrecipesagainstmultipleplatformslocallyDeveloprecipesfasterwithlocaltestexecutionbeforerunningthemonahostCatchthechangesindependenciesthatwillotherwisebreakyourinfrastructurebeforetheygetdeployedWritetestsforbugstopreventthemfromhappeningagaininthefuture(regression)

TestingrecipesThereareanumberofwaystotestyourrecipes.Oneapproachistosimplyfollowtheprocessofdevelopingyourrecipes,uploadingthemtoyourChefserver,anddeployingthemtoahost;repeatthisuntilyouaresatisfied.Thishasthebenefitofexecutingyourrecipesonrealinstances,butthedrawbackisthatitisslow,particularlyifyouaretestingonmultipleplatforms,andrequiresthatyoumaintainafleetofhosts.Ifyourcookbookruntimesarereasonablyshortandyouhaveasmallnumberofplatformstosupportthem,thenthismightbeaviableoption.Thereisabetteroptiontotestyourrecipes,anditiscalledChefSpec.ForthosewhohaveusedRSpec,aRubytestinglibrary,theseexampleswillbeanaturalextensionofRSpec.IfyouhaveneverusedRSpec,thebeginningofthischapterwillintroduceyoutoRSpec’stestinglanguageandmechanisms.

RSpecRSpecisaframeworktotestRubycodethatallowsyoutouseadomain-specificlanguagetoprovidetests,muchinthesamewayChefprovidesadomain-specificlanguagetomanipulateaninfrastructure.InsteadofusingaDSLtomanagesystems,RSpec’sDSLprovidesanumberofcomponentstoexpresstheexpectationsofcodeandsimulatetheexecutionofportionsofthesystem(alsoknownasmocking).

ThefollowingexamplesinRSpecshouldgiveyouahigh-levelideaofwhatRSpeccando:

#simpleexpectation

it'shouldadd2and2together'do

x=2+2

expect(x).toeq4

end

#EnsureanyinstanceofObjectreceivesacallto'foo'

#andreturnapre-definedvalue(mocking)

it'verifiesthataninstancereceives:foo'do

expect_any_instance_of(Object)

.toreceive(:foo).and_return(:return_value)

o=Object.new

expect(o.foo).toeq(:return_value)

end

#Deepexpectations(i.eclientmakesanHTTPcallsomewhere

#insideit,makesureithappensasexpected)

it'shouldmakeanauthorizedHTTPGETcall'do

expect_any_instance_of(Net::HTTP::Get)

.toreceive(:basic_auth)

@client.make_http_call

end

RSpecandChefSpecAswithmosttestinglibraries,RSpecenablesyoutoconstructasetofexpectations,buildobjectsandinteractwiththem,andverifythattheexpectationshavebeenmet.Forexample,oneexpectsthatwhenauserlogsintothesystem,adatabaserecordiscreated,trackingtheirloginhistory.However,tokeeptestsrunningquickly,theapplicationshouldnotmakeanactualdatabasecall;inplaceoftheactualdatabasecall,amockmethodshouldbeused.Here,ourmockmethodwillcatchthemessageinthedatabaseinordertoverifythatitwasgoingtobesent;then,itwillreturnanexpectedresultsothatthecodedoesnotknowthedatabaseisnotreallythere.

TipMockmethodsaremethodsthatareusedtoreplaceonecallwithanother;youcanthinkofthemasstuntdoubles.Forexample,ratherthanmakingyourcodeactuallyconnecttothedatabase,youmightwanttowriteamethodthatactsasthoughithassuccessfullyconnectedtothedatabaseandfetchedtheexpecteddata.

ThiscanbeextendedtomodelChef’sabilitytohandlemultipleplatformsandenvironmentsverynicely;codeshouldbeverifiedtobehaveasexpectedonmultipleplatformswithouthavingtoexecuterecipesonthoseplatforms.ThismeansthatyoucantesttheexpectationsaboutRedHatrecipesfromanOSXdevelopmentmachineorWindowsrecipesfromanUbuntudesktop,withoutneedingtohavehostsaroundtodeploytofortestingpurposes.Additionally,thedevelopmentcycletimeisgreatlyreducedastestscanbeexecutedmuchfasterwithexpectationsthanwhentheyareperformingsomeworkonanendhost.

Youmaybeaskingyourself,“Howdoesthisreplacetestingonanactualhost?”Theansweristhatitmaynot,andsoyoushoulduseintegrationtestingtovalidatethattherecipesworkwhendeployedtorealhosts.Whatitdoesallowyoutodoisvalidateyourexpectationsofwhatresourcesarebeingexecuted,whichattributesarebeingused,andthatthelogicalflowofyourrecipesarebehavingproperlybeforeyoupushyourcodetoyourhosts.Thisformsatighterdevelopmentcycleforrapiddevelopmentoffeatureswhileprovidingalonger,moreformallooptoensurethatthecodebehavescorrectlyinthewild.

Ifyouarenewtotestingsoftware,andinparticular,testingRubycode,thisisabriefintroductiontosomeoftheconceptsthatwewillcover.Testingcanhappenatmanydifferentlevelsofthesoftwarelifecycle:

Single-modulelevel(calledunittests)Multi-modulelevel(knownasfunctionaltests)System-leveltesting(alsoreferredtoasintegrationtesting)

TestingbasicsInthetest-driven-development(TDD)philosophy,testsarewrittenandexecutedearlyandoften,typically,evenbeforecodeiswritten.Thisguaranteesthatyourcodeconformstoyourexpectationsfromthebeginninganddoesnotregresstoapreviousstateofnon-conformity.ThischapterwillnotdiveintotheTDDphilosophyandcontinuoustesting,butitwillprovideyouwithenoughknowledgetobegintestingtherecipesthatyouwriteandfeelconfidentthattheywilldothecorrectthingwhendeployedintoyourproductionenvironment.

ComparingRSpecwithothertestinglibrariesRSpecisdesignedtoprovideamoreexpressivetestinglanguage.ThismeansthatthesyntaxofanRSpectest(alsoreferredtoasaspectestorspec)isdesignedtocreatealanguagethatfeelsmorelikeanaturallanguage,suchasEnglish.Forexample,usingRSpec,onecouldwritethefollowing:

expect(factorial(4)).toeq24

Ifyoureadtheprecedingcode,itwillcomeoutlikeexpectfactorialof4toequal24.ComparethistoasimilarJUnittest(forJava):

assertEquals(24,factorial(4));

Ifyoureadtheprecedingcode,itwouldsoundmorelikeassertthatthefollowingareequal,24andfactorialof4.Whilethisisreadablebymostprogrammers,itdoesnotfeelasnaturalastheonewesawearlier.

RSpecalsoprovidescontextanddescribeblocksthatallowyoutogrouprelatedexamplesandsharedexpectationsbetweenexamplesinthegrouptohelpimproveorganizationandreadability.Forexample,considerthefollowingspectest:

describeArraydo

it"shouldbeemptywhencreated"do

Array.new.should==[]

end

end

ComparetheprecedingtesttoasimilarNUnit(.NETtestingframework)test:

namespaceMyTest{

usingSystem.Collection

usingNUnit.Framework;

[TestFixture]

publicclassArrayTest{

[Test]

publicvoidNewArray(){

ArrayListlist=newArrayList();

Assert.AreEqual(0,list.size());

}

}

}

Clearly,thespectestismuchmoreconciseandeasiertoread,whichisagoalofRSpec.

UsingChefSpecChefSpecbringstheexpressivenessofRSpectoChefcookbooksandrecipesbyprovidingChef-specificprimitivesandmechanismsontopofRSpec’ssimpletestinglanguage.Forexample,ChefSpecallowsyoutosaythingslike:

it'createsafile'do

expect(chef_run).tocreate_file('/tmp/myfile.txt')

end

Here,chef_runisaninstanceofafullyplannedChefclientexecutiononadesignatedendhost,aswewillseelater.Also,inthiscase,itisexpectedthatitwillcreateafile,/tmp/myfile.txt,andthetestwillfailifthesimulatedrundoesnotcreatesuchafile.

GettingstartedwithChefSpecInordertogetstartedwithChefSpec,createanewcookbookdirectory(hereitis$HOME/cookbooks/mycookbook)alongwitharecipesandspecdirectory:

mkdir-p~/cookbooks/mycookbook

mkdir-p~/cookbooks/mycookbook/recipes

mkdir-p~/cookbooks/mycookbook/spec

Nowyouwillneedasimplemetadata.rbfileinsideyourcookbook(here,thiswillbe~/cookbooks/mycookbook/metadata.rb):

maintainer"Yournamehere"

maintainer_email"you@domain.com"

license"Apache"

description"Simplecookbook"

long_description"Supersimplecookbook"

version"1.0"

supports"debian"

Oncewehavethis,wenowhavethebarebonesofacookbookthatwecanbegintoaddrecipesandteststo.

InstallingChefSpecInordertogetstartedwithChefSpec,youwillneedtoinstallagemthatcontainstheChefSpeclibrariesandallthesupportingcomponents.Notsurprisingly,thatgemisnamedchefspecandcanbeinstalledsimplybyrunningthefollowing:

geminstallchefspec

However,becauseRubygemsoftenhaveanumberofdependencies,theRubycommunityhasbuiltatoolcalledBundlertomanagegemversionsthatneedtobeinstalled.SimilartohowRVMprovidesinterpreter-levelversionmanagementandawaytokeepyourgemsorganized,Bundlerprovidesgem-levelversionmanagement.WewilluseBundlerfortworeasons.Inthiscase,wewanttolimitthenumberofdifferencesbetweentheversionsofsoftwareyouwillbeinstallingandtheversionstheauthorhasinstalledtoensurethatthingsareassimilaraspossible;secondly,thisextendswelltoreleasingproductionsoftware—limitingthenumberofvariablesiscriticaltoconsistentandreliablebehavior.

LockingyourdependenciesinRubyBundlerusesafile,specificallynamedGemfile,todescribethegemsthatyourprojectisdependentupon.Thisfileisplacedintherootofyourproject,anditscontentsinformBundlerwhichgemsyouareusing,whatversionstouse,andwheretofindgemssothatitcaninstallthemasneeded.

Forexample,hereistheGemfilethatisbeingusedtodescribethegemversionsthatareusedwhenwritingtheseexamples:

source'https://rubygems.org'

gem'chef','11.10.0'

gem'chefspec','3.2.0'

gem'colorize','0.6.0'

Usingthiswillensurethatthegemsyouinstalllocallymatchtheonesthatareusedwhenwritingtheseexamples.Thisshouldlimitthedifferencesbetweenyourlocaltestingenvironmentsifyouruntheseexamplesonyourworkstation.

InordertouseaGemfile,youwillneedtohaveBundlerinstalled.IfyouareusingRVM,Bundlershouldbeinstalledwitheverygemsetyoucreate;ifnot,youwillneedtoinstallitonyourownviathefollowingcode:

geminstallbundler

OnceBundlerisinstalledandaGemfilethatcontainsthepreviouslinesisplacedintherootdirectoryofyourcookbook,youcanexecutebundleinstallfrominsideyourcookbook’sdirectory:

user@host:~/cookbooks/mycookbook$>bundleinstall

BundlerwillparsetheGemfileinordertodownloadandinstalltheversionsofthegemsthataredefinedinside.Here,Bundlerwillinstallchefspec,chef,andcolorizealongwithanydependenciesthosegemsrequirethatyoudonotalreadyhaveinstalled.

CreatingasimplerecipeandamatchingChefSpectestOncethesedependenciesareinstalled,youwillwanttocreateaspectestinsideyourcookbookandamatchingrecipe.InkeepingwiththeTDDphilosophy,wewillfirstcreateafile,default_spec.rb,inthespecdirectory.Thenameofthespecfileshouldmatchthenameoftherecipefile,onlywiththeadditionof_specattheend.Ifyouhavearecipefilenameddefault.rb(whichmostcookbookswill),thematchingspectestwouldbecontainedinafilenameddefault_spec.rb.Let’stakealookataverysimplerecipeandamatchingChefSpectest.

WritingaChefSpectestThetest,shownasfollows,willverifythatourrecipewillcreateanewfile,/tmp/myfile.txt:

require'chefspec'

describe'mycookbook::default'do

let(:chef_run){

ChefSpec::Runner.new.converge(described_recipe)

}

it'createsafile'do

expect(chef_run).tocreate_file('/tmp/myfile.txt')

end

end

Here,RSpecusesadescribeblocksimilartothewayChefusesaresourceblock(again,blocksareidentifiedbythedo…endsyntaxorcodecontainedinsidecurlybraces)todescribearesource,inthiscase,thedefaultrecipeinsideofmycookbook.Thedescribedresourcehasanumberofexamples,andeachexampleisdescribedbyanitblocksuchasthefollowing,whichcomesfromthepreviousspectest:

it'createsafile'do

expect(chef_run).tocreate_file('/tmp/myfile.txt')

end

Thestringgiventotheitblockprovidestheexamplewithahuman-readabledescriptionofwhattheexampleistesting;inthiscase,weareexpectingthattherecipecreatesafile.WhenourrecipesarerunthroughChefSpec,theresourcesdescribedarenotactuallycreatedormodified.Instead,amodelofwhatwouldhappenisbuiltastherecipesareexecuted.ThismeansthatChefSpeccanvalidatethatanexpectedactionwouldhaveoccurrediftherecipeweretobeexecutedonanendhostduringarealclientrun.

TipIt’simportanttonotethateachexampleblockresetsexpectationsbeforeitisexecuted,soanyexpectationsdefinedinsideofagiventestwillnotfallthroughtoothertests.

BecausemostofthetestswillinvolvesimulatingaChefclientrun,wewanttorunthe

simulationeverytime.Therearetwooptions:executethecodeineveryexampleoruseasharedresourcethatallthetestscantakeadvantageof.Inthefirstcase,thetestwilllooksomethinglikethefollowing:

it'createsafile'do

chef_run=ChefSpec::Runner.new.converge(described_recipe)

expect(chef_run).tocreate_file('/tmp/myfile.txt')

end

Theprimaryproblemwiththisapproachisrememberingthateverytestwillhavetohavetheresourcerunningatthebeginningofthetest.Thistranslatestoalargeamountofduplicatedcode,andiftheclientneedstobeconfigureddifferently,thenthecodeneedstobechangedforallthetests.Tosolvethisproblem,RSpecprovidesaccesstoasharedresourcethroughabuilt-inmethod,let.Usingletallowsatesttodefineasharedresourcethatiscachedforeachexampleandresetasneededforthefollowingexamples.Thisresourceisthenaccessibleinsideofeachblockasalocalvariable,andRSpectakescareofknowingwhentoinitializeitasneeded.

Ourexampletestusesaletblocktodefinethechef_runresource,whichisdescribedasanewChefSpecrunnerforthedescribedrecipe,asshowninthefollowingcode:

let(:chef_run){

ChefSpec::Runner.new.converge(described_recipe)

}

Here,described_recipeisaChefSpecshortcutforthenameoftherecipeprovidedinthedescribeblock.Again,thisisaDRY(don’trepeatyourself)mechanismthatallowsustorenametherecipeandthenonlyhavetochangethenameofthedescriptionratherthanhuntthroughthecode.Thesetechniquesmaketestsbetterabletoadapttochangesinnamesandresources,whichreducescoderotastimegoesby.

BuildingyourrecipeTherecipe,asdefinedhere,isaverysimplerecipewhoseonlyjobistocreateasimplefile,/tmp/myfile.txt,ontheendhost:

file"/tmp/myfile.txt"do

owner"root"

group"root"

mode"0755"

action:create

end

Putthisrecipeintotherecipes/default.rbfileofyourcookbooksothatyouhavethefollowingfilelayout:

mycookbook/

|-recipes/

||-default.rb

|-spec/

|-default_spec.rb

Executingtests

Inordertorunthetests,weusetherspecapplication.ThisisaRubyscriptthatcomeswiththeRSpecgem,whichwillrunthetestscriptsasspectestsusingtheRSpeclanguage.ItwillalsousetheChefSpecextensionsbecauseinourspectest,wehaveincludedthemviathelinerequire'chefspec'atthetopofourdefault_spec.rbfile.Here,rspecisexecutedthroughBundlertoensurethatthedesiredgemversions,asspecifiedinourGemfile,areusedatruntimewithouthavingtoexplicitlyloadthem.Thisisdoneusingthebundleexeccommand:

bundleexecrspecspec/default_spec.rb

ThiswillrunRSpecusingBundlerandprocessthedefault_spec.rbfile.Asitruns,youwillseetheresultsofyourtests,a.(period)forteststhatpass,andanFforanyteststhatfail.Initially,theoutputfromrspecwilllooklikethis:

Finishedin0.17367seconds

1example,0failures

RSpecsaysthatitcompletedtheexecutionin0.17secondsandthatyouhadoneexamplewithzerofailures.However,theresultswouldbequitedifferentifwehaveafailedtest;RSpecwilltelluswhichtestfailedandwhy.

UnderstandingfailuresRSpecisverygoodattellingyouwhatwentwrongwithyourtests;itdoesn’tdoyouanygoodtohavefailingtestsifit’simpossibletodeterminewhatwentwrong.Whenanexpectationinyourtestisnotmet,RSpecwilltellyouwhichexpectationwasunmet,whattheexpectedvaluewas,andwhatvaluewasseen.

Inordertoseewhathappenswhenatestfails,modifyyourrecipetoensurethatthetestfails.Lookinyourrecipeforthefollowingfileresource:

file"/tmp/myfile.txt"do

Replacethefileresourcewithadifferentfilename,suchasmyfile2.txt,insteadofmyfile.txt,likethefollowingexample:

file"/tmp/myfile2.txt"do

Next,rerunyourspectests;youwillseethatthetestisnowfailingbecausethesimulatedChefclientexecutiondidsomethingthatwasunexpectedbyourspectest.Anexampleofthisnewexecutionwouldlooklikethefollowing:

[user@host]$bundleexecrspecspec/default_spec.rb

F

Failures:

1)my_cookbook::defaultcreatesafile

Failure/Error:expect(chef_run).tocreate_file('/tmp/myfile.txt')

expected"file[/tmp/myfile.txt]"withaction:createtobeinChef

run.Otherfileresources:

file[/tmp/myfile2.txt]

#./spec/default_spec.rb:9:in`block(2levels)in<top(required)>'

Finishedin0.18071seconds

1example,1failure

Noticethatinsteadofadot,thetestresultsinanF;thisisbecausethetestisnowfailing.Asyoucanseefromthepreviousoutput,RSpecistellingusthefollowing:

Thecreatesafileexampleinthe'my_cookbook::default'testsuitefailedOurexamplefailedintheninthlineofdefault_spec.rb(asindicatedbythelinethatcontained./spec/default_spec.rb:9)Thefileresource/tmp/myfile.txtwasexpectedtobeoperatedonwiththe:createactionTherecipeinteractedwithafileresource/tmp/myfile2.txtinsteadof/tmp/myfile.txt

RSpecwillcontinuetoexecuteallthetestsinthefilesspecifiedonthecommandline,printingouttheirstatusastowhethertheypassedorfailed.Ifyourtestsarewellwrittenandruninisolation,thentheywillhavenoeffectononeanother;itshouldbesafetoexecuteallofthemevenifsomefailsothatyoucanseewhatisnolongerworking.

ExpandingyourtestsChefSpecprovidesacomprehensivesuiteoftoolstotestyourrecipes;youcanstubandmockresources(replacerealbehaviorwithartificialbehavior,suchasnetworkordatabaseconnections),simulatedifferentplatforms,andmore.Let’stakealookatsomemorecomplexexamplestoseewhatotherthingswecandowithChefSpec.

MultipleexamplesinaspectestSpectestsdonotneedtocontainonlyoneexample;theycancontainasmanyasyouneed.Inordertoorganizethem,youcangroupthemtogetherbywhattheydescribeandsomesharedcontext.InRSpec,contextblockscontainexamplesthatarerelevanttotherecipeorscriptbeingtested.Thinkofthemasself-containedtestsuiteswithinalargertestsuite;theycanhavetheirownresourcesaswellassetupandtear-downlogicthatarespecifictotheteststhatareruninthatcontext.

Asanexample,let’slookatpartofthespectestsuitefromtherender_fileexampleinsideofChefSpecitself.Considerthisportionoftherecipe:

file'/tmp/file'do

content'Thisiscontent!'

end

cookbook_file'/tmp/cookbook_file'do

source'cookbook_file'

end

template'/tmp/template'do

source'template.erb'

end

Therecipebeingshownhasthreeresources:atemplate,acookbook_file,andanordinaryfileresource.Asampleofthematchingspectest(testsremovedforformattingandeaseofreading)containsanouterdescribeblock,whichtellsusthatweareexecutingtestsfortherender_file::defaultrecipeandthreeseparatecontextblocks.Eachcontextdescribesadifferentportionoftherecipethatisbeingtestedandtheexpectationsofthatparticulartypeofresource.Together,theyareallpartofthedefaultrecipe,buttheybehaveverydifferentlyinwhatcontenttheyrenderaswellaswhereandhowtheystorefilesonthesystem.

Inthisexample,thefilecontextcontainsteststhatpertaintotheexpectedresultsofthefileresource,thecookbook_filecontextisconcernedwiththecookbook_fileresource,andsoon:

describe'render_file::default'do

let(:chef_run){

ChefSpec::Runner.new.converge(described_recipe)

}

context'file'do

it'rendersthefile'do

expect(chef_run).torender_file('/tmp/file')

expect(chef_run).to_notrender_file('/tmp/not_file')

end

end

context'cookbook_file'do

it'rendersthefile'do

expect(chef_run).torender_file('/tmp/cookbook_file')

expect(chef_run).to_not

render_file('/tmp/not_cookbook_file')

end

end

context'template'do

it'rendersthefile'do

expect(chef_run).torender_file('/tmp/template')

expect(chef_run).to_notrender_file('/tmp/not_template')

end

end

end

Contextscanbeusedtogrouptogetherasetofexamplesthatarerelated,notjustonesthatarespecifictoaparticularresource.Considerthefollowingexample:

describe'package::install'do

context'wheninstallingonWindows2012'do

end

context'wheninstallingonDebian'do

end

context'wheninstallingonFreeBSD'do

end

end

Inthepreviousexample,ourspectestcontainedteststhataregroupedtogetherbytheplatformbeingexecutedon.Insideofeachcontext,theChefrunwillbeconstructedwithaplatformargumentinsteadsothattheexpectationsbeingtestedwillbeconsideredagainstarunoftherecipeontheplatforminquestionratherthanthehost’soperatingsystem.Thisisincrediblyuseful,aswewillseeinthenextsectionontestingformultipleplatforms.

TestingformultipleplatformsOneofthemorenon-trivialusesofChefSpecistosimulateexecutingrecipesonmultipleplatforms.Thisisusefulfordeveloperswhoarebuildingrecipesthatneedtosupportmorethanoneoperatingsystem.SoftwarepackagessuchasPostgreSQL,MySQL,Java,PHP,Apache,andcountlessotherapplicationscanbeinstalledonmanydifferentplatforms.Becauseeachplatformvariesinitsinstallationmechanism,usercreation,andothercorefeatures,beingabletotestrecipesagainstallthesupportedplatformsisincrediblyuseful.

Let’slookatahypotheticalexampletodeveloparecipetoinstallMySQLonWindows2012andsomethingswemightwanttovalidateduringsucharun:

context'whenrunonWindows2012'do

let(:chef_run)do

#constructa'runner'(simulatechef-client)running

#onaWindows2012host

runner=ChefSpec::ChefRunner.new(

'platform'=>'windows',

'version'=>'2012'

)

#setaconfigurationvariable

runner.node.set['mysql']['install_path']='C:\\temp'

runner.node.set['mysql']['service_user']='SysAppUser'

runner.converge('mysql::server')

end

it'shouldincludethecorrectWindowsserverrecipe'do

chef_run.shouldinclude_recipe'mysql::server_windows'

end

it'shouldcreateanINIfileintherightdirectory'do

ini_file="C:\\temp\\mysql\\mysql.ini"

expect(chef_run).tocreate_templateini_file

end

end

Byconstructingtherunnerwiththeplatformandversionoptions,thetestwillexerciserunningthemysql::serverrecipeandpretendasthoughitwererunningonaWindows2012host.Thisallowsustosetupexpectationsaboutthetemplatesthatwillbecreated,recipesthatarebeingexecuted,andmoreonthatparticularplatform.

Presumingthatthemysql::serverrecipewasabletodelegatetotheOS-specificrecipeonagivenplatform,wecouldwriteanothertest:

context'whenrunonDebian'do

let(:chef_run)do

runner=ChefSpec::ChefRunner.new(

'platform'=>'debian'

)

runner.node.set['mysql']['install_path']='/usr/local'

runner.node.set['mysql']['service_user']='mysql'

runner.converge('mysql::server')

end

it'shouldincludethecorrectLinuxserverrecipe'do

chef_run.shouldinclude_recipe'mysql::server_linux'

end

it'shouldcreateanINIfileintherightdirectory'do

ini_file="/usr/local/mysql/mysql.ini"

expect(chef_run).tocreate_templateini_file

end

it'shouldinstalltheDebianMySQLpackage'do

expect(chef_run).toinstall_package('mysql-server')

end

end

Inthisway,wecanwriteourteststovalidatetheexpectedbehavioronplatformsthatwemaynothavedirectaccesstoinordertoensurethattheywillbeperformingtheexpected

actionsforacollectionofplatforms.

SummaryRSpecwithChefSpecextensionsprovidesuswithincrediblypowerfultoolstotestourcookbooksandrecipes.YouhaveseenhowtodevelopbasicChefSpectestsforyourrecipes,organizeyourspectestsinsideofyourcookbook,executeandanalyzetheoutputofyourspectests,andsimulatetheexecutionofyourrecipesacrossmultipleplatforms.

Infuturechapters,wewilllearnsomemoreadvancedtestingmechanismssuchasmockingandstubbingexternalservicessuchassearchanddatabags.Addingtestingtoyourdevelopmentcycleallowsyoutofeelconfidentinthecorrectnessofyourrecipes,whichisacriticalsteptowardsautomatingthemanagementofyourinfrastructure.

Now,let’stakealookathowwewillbuildacookbooktocomplementawebapplicationsothatwecanseethefullcycleofdevelopinganapplicationanddeployingitusingChef.

Chapter6.FromDevelopmenttoDeploymentThischaptercoversend-to-endsoftwaredeploymentofaPython-basedwebapplication.Itwillalsointroduceyoutosomecommoncookbooksandhowtoputthemalltogethertocreateafullyautomateddeploymentmechanism.

Wewillwalkthroughthefollowingtopics:

ConfiguringyourlocalsettingstoworkwithAWSModelingasimpleweb.pyapplicationwithChefInstallingthecookbooksyouneedProvisioningEC2instancesforwebandDBserversDefiningyourrolesAddinguserstohostsInstallingtherequiredsoftwareConfiguringanapplicationusingChefDeployingtheapplication

DescribingthesetupFromahighlevel,hereiswhatneedstohappeninordertotakeanapplicationthatwehavedevelopedfromadesktoptodeployment.Inordertodeployyourapplication,youwillprovisiontwohosts,webanddb(eachwithoneuseraccount)andwebapp,whosehomedirectorywillbein/home/webapp.ThesourcecodewillbehostedonGitHubanddeployedusingGitontothewebserver.Wewillcreateadatabase,provisionanaccounttoaccessthatdatabase,andthenconfigureanddeployaweb.pyapplicationinavirtualPythonenvironmentthatwillbestartedandmonitoredbysupervisord.Thisisafairlycommonpatternformodernwebapplications,regardlessoftheframeworkandlanguagebeingused.Thedemonstrationapplicationusedinthischapterconsistsofonlyahandfuloffilesmakingiteasytodeployandunderstand,butthiswillgiveyoutheconceptsandtoolstoexpandthisexampleforusewithfutureapplicationsyoumightdeveloporneedtodeploy.

DeployingsoftwarewithChefTherearenumerousbenefitstodeployingyoursoftwareusingChef;theprimarybenefitisautomation—thechef-clientcanberunperiodically,anditcanexecutefully-automateddeploymentswheneverchangesaremadetothesourcecoderepository.Additionally,Chefstoresallyourconfigurationdata,soyoucanavoidstoringsensitivesecretsandhard-codingURLsorotherdynamicdatainyourconfiguration.Forexample,ifyouhaveanapplicationwithadatabasepool,andyouaddanewdatabasehosttoyourpool,Chefcanuseasimplesearchtopopulatethelistofhoststoincludeintheconnectionpoolsothatitisalwaysuptodatewithyourinfrastructure.

However,deployingsoftwarewithChefdoesrequiresomecoordinationbetweenyourapplicationandChef.Youwillneedtomaintainrecipesrequiredfordeployingyourapplication,andyouwillalsowanttouseChefastheauthoritativesourceforyourconfigurationdata,whichinvolveswritingconfigurationtemplates.ByusingCheftomanageyourdeployments,youcanalsogenerateanyconfigurationdataneededtorunyoursoftwarebasedonyourinfrastructureconfiguration;inourcase,asimpleconfig.pyfileforyourweb.pyapplication.Thismethodcanalsobeusedtomanagethedatabase.ymlfile(andanyotherYAMLfiles)forRailsapplications,theserver.ymlfileforaDropwizardapplication,oranyotherconfigurationfilesneededtorunyourservice.

TipYAMLisasimplemarkuplanguagetostoreconfigurationdata.Itispopularwithmoderndevelopersbecauseitiseasytoparseandisveryexpressive,similartoJSON.

ConfiguringyourlocalenvironmentBynow,youshouldhaveaccesstoaChefserverofsomesort;here,wewillbeusingthehostedChefservice,butthework(asidefromconfiguringyourknife.rbfile)willremainthesameacrossself-managedandhostedChefinstances.Inordertofollowalongwiththeexamples,youwillneedtoconfigureyourworkstationwithanappropriateknife.rbfileandcertificates.ThesefilescanbedownloadedfromthehostedChefconsoleandmodifiedasneeded.

Additionally,youwillneedtohaveinstalledknifethroughthechefgemandhavetheknife-ec2geminstalledinordertointeractwithEC2.Ifyouprefertouseadifferentprovider,thenyoucanrefertothepreviouschapteronhowtoprovisioncloudhostswiththisproviderwhenyougettotheprovisioningstep.

Inourexample,usinghostedChefandEC2,ourknife.rbfilewillcontaincontentsimilartothefollowingcode:

current_dir=File.dirname(__FILE__)

log_level:info

log_locationSTDOUT

node_name"myorg"

client_key"#{current_dir}/myorg.pem"

validation_client_name"myorg-validator"

validation_key"#{current_dir}/myorg-validator.pem"

chef_server_url"https://api.opscode.com/organizations/myorg"

cache_type'BasicFile'

cache_options(:path=>"#{ENV['HOME']}/.chef/checksums")

cookbook_path["#{current_dir}/../cookbooks"]

knife[:aws_access_key_id]="YOURAWSACCESSKEY"

knife[:aws_secret_access_key]="YOURAWSSECRETKEY"

knife[:region]="AWSREGION"

Again,thebasefilescanbedownloadedfromthehostedChefconsole,orifyouareusingaself-managedChefinstallation,thiscanbefoundonyourChefserver.

ModelingasimplePythonapplicationHerewewillconsideraweb.pyapplicationthathastwoprimarycomponents:awebserverandadatabaseserver.Wewillprovisiononehostforeachrole,bootstrapthem,anddeploythesoftwareontoournewhosts.

Ourapplicationstackwillconsistofthefollowing:

web.pyasourwebframeworkPostgreSQLfordatastorageEC2forvirtualhosts

Wewanttodefinetwoprimaryrolesthatrepresentourwebserverandourdatabaseserver.Inaddition,wewillconstructabaselineroleforallourserversthatwillsupplyanycommondataweneedsuchasuseraccounts,SSHkeys,networkconfigurationdata,shells,commonutilities,libraries,andsoon.

Wewillneedtofindorwritecookbooksforthefollowingcomponentswewilluse:

PythonsupervisordPostgreSQLUseraccountsOurcustomwebapplication

ManagingthecookbooksThecookbooksthatwewillbeusingareallavailableatthefollowingURL:https://github.com/johnewart/simplewebpy_app.Becauseanumberofcookbooksusedinthisexampleareunderactivedevelopment,theonesrequiredfortheexampleshavebeenfrozen(asofwritingofthisbook)toensurecompatibilitywiththeexamples;itisbettertohavethemslightlyoutofdatethanbrokeninthiscase.

However,whenyouwriteyourowncookbooksanddeployyourownsoftwarebeyondthisexample,youwillfindthattherearealargenumberofcookbooksthatcanbefoundthroughtheChefcommunitysite—http://supermarket.getchef.com/—orbysearchingtheWebforcookbooks;manyofthesewillbehostedonGitHub,BitBucket,orsimilarsourcecode-hostingsites.

DownloadingcookbooksHereinthefollowingcode,wewillsimplydownloadthecookbookcollectionasawhole:

http://github.com/johnewart/chef_essential_files

Toinstallthecollection,wecandothefollowingfromthechef_essential_files/cookbooksdirectory:

knifecookbookinstall-o.*

Thiswillinstallallofthecookbooksthatareprovided.Theprovidedcookbooksareallthatisrequiredfortheexamplesinthischaptertobesuccessful.Let’stakealookatourcustomcookbook,thepythonwebappcookbook,asalloftheothersareoff-the-shelfcookbooksthataredesignedtoprovidesomegeneralsupportfunctionalities.

LookingatthedatabaserecipeWewilldoafewthingshere,solet’slookatourdatabaserecipe.Inorderforourwebapplicationtobeuseful,itneedsadatabasetoconnectto.Typically,thisinvolvesinstallingthedatabaseserversoftware,constructingadatabase,andgrantingaccesstothatdatabasebyaspecifieduser(orusers).Ourapplicationisnodifferent,sowewillleveragethedatabasecookbookinordertoaccomplishthis.

First,inourrecipe,weneedtoincludethePostgreSQL-specificresourcesfromthedatabasecookbook,whichwewilldousingthefollowingcode:

include_recipe"database::postgresql"

Youwillneedtoknowwhatdatabaseyouwillbecreatingandtowhichuseryouwillbegrantingaccesstoalongwiththepasswordthatwillbeusedtoidentifythem:

dbname=node[:webapp][:dbname]

dbuser=node[:webapp][:dbuser]

dbpass=node[:webapp][:dbpass]

Inordertocreateadatabaseanduseraswellasgrantaccess,youwillneedtoestablishaconnectiontothedatabaseserverwithauserthathaspermissiontodoso.Youwillseethatthisuserhasalsobeengrantedaccessinyourrole’spg_hbasettingssothatPostgreSQLknowstoallowthepostgresusertoconnecttothedatabaselocally,asshowninthefollowingcode:

postgresql_connection_info={

:host=>'localhost',

:port=>node['postgresql']['config']['port'],

:username=>'postgres',

:password=>node['postgresql']['password']['postgres']

}

Usingthisconnectioninformation,youcanconstructadatabaseandauser(iftheydon’talreadyexist),andthengrantthatuserfullaccesstoournewdatabase:

#Constructanactualdatabaseontheserver

postgresql_database'webapp'do

connectionpostgresql_connection_info

action:create

end

#Createadatabaseuserresourceusingourconnection

postgresql_database_userdbuserdo

connectionpostgresql_connection_info

passworddbpass

action:create

end

#Grantallprivilegesonalltablesin'webapp'

postgresql_database_userdbuserdo

connectionpostgresql_connection_info

database_namedbname

privileges[:all]

action:grant

end

Thishigh-levellanguageallowsustoeasilymanipulatethedatabasewithouttheneedtoknowanydatabase-specificSQLorcommands.IfyouwanttoconvertyourapplicationtouseMySQL,forexample,provisioninganewMySQLdatabasewouldlargelybeaseasyasconvertingthewordpostgresqltomysqlinourrecipe,andthedatabase-specificadapterinthedatabasecookbookwillberesponsiblefortheimplementationdetails.

LookingatyourapplicationdeploymentcookbookOnceourdatabasehasbeenprovisioned,youcanlookathowyoucaninstallourwebapplication.Inthepythonwebapp::webapprecipe,youhavealltheinformationyouneedtodothis.Thewaythatyoudefinearecipefordeployinganapplicationwillvarywildlyamongapplications,aseachapplicationisunique.However,thisparticularexamplewasdesignedtobearepresentativeofmostwebapplications(reasonably)andshouldpresentyouwithagoodstartingpointtounderstandthebasicsofdeployingawebapplicationwithChef.

Modernwebapplicationstypicallyfollowthesamepattern:provisionauser,installaninterpreter,orotherengine(suchasPython,Ruby,andJava),createdirectoriesifneeded,checkoutthesourcecode,runanydatamigrations(ifneeded)toupdateyourdatabase,andthenmakesurethatyourserviceisupandrunning;thisisnodifferent.Themorecomplicatedyourapplication,themoreinfrastructureyoumayneedtomodel,suchasjobqueueengines,asynchronousworkers,andotherlibraries.

Ifyoulookatthewebapplicationcookbooklocatedatcookbooks/pythonwebapp,youwillseethatithasthefollowing:tworecipes,atemplate,andaPIP-requirementdefinitioninsideit.TherecipesincludedareforthewebapplicationitselfandtomanagethecreationofthePostgreSQLdatabaseanduseronthedatabasehost.

Mostoftheinterestingworkisintheapplicationrecipe,cookbooks/pythonwebapp/recipes/webapp.rb;so,let’sstartbytakingalookatthat.Allapplicationsaregoingtohaveaslightlydifferentdeploymentlogic,butmodernwebapplicationsusuallyfollowapatternthatlookslikethefollowing:

Installanysystem-widepackagesrequiredConstructthedirectoriesneededforthesoftwareIfthisisPythonorRuby,possiblyinstallanewvirtualenvtoolorRVMgemsetInstallthelibrariesneededtoruntheapplicationCheckouttheapplication’ssourcecodeBuildandconfiguretheapplicationasneededCreateorupdatethedatabaseschemaStopthewebapplicationservicesStartthewebserverorprocessmanagerthatmonitorstheapplication

Thisexampleapplicationisnodifferent,solet’slookatthestepsneededtodeploythisweb.pyapplication.First,declareanyapplicationconfigurationdataneededwiththefollowingcommand:

app_root=node[:webapp][:install_path]

python_interpreter=node[:webapp][:python]

virtualenv="#{app_root}/python"

virtual_python="#{virtualenv}/bin/python"

src_dir="#{app_root}/src/"

#Grabthefirstdatabasehost

dbhost=search(:node,"role:postgresql_server")[0]['ipaddress']

environment_hash={

"HOME"=>"#{app_root}",

"PYTHONPATH"=>"#{app_root}/src"

}

Inthissnippet,weusedthecomputedattributestotellourrecipewheretoinstalltheapplication;inthiscase,thedefaultis/opt/webappbutthiscanbeoverriddenforflexibility.Additionally,wesetthepathtothePythoninterpreterwewanttouseforourPythonvirtualenv.However,youcanjustaseasilyspecifyaRubyorJavapathifyourapplicationusedoneofthoselanguages.Thereisapathtothesourcecodeandadatabasehostaddress.ThispathisdeterminedbysearchingtheChefdataforallnodeswiththepostgresql_serverrole,takingthefirstone,andusingitsIP.Thisallowsustoreplacethedatabaseserverandnothavetoworryaboutupdatingourconfigurationdata,whichwe’llseeinabit.

PreparingthedirectoriesInordertodeployourapplication,andforittorun,weneedtohavealocationtoputourdata.Inthisapplication,wehavedefinedaneedfor:aconfigurationdirectory,alogdirectory,andaplacetoviewthesourcecode.Inourrecipe,wewillcreatethesedirectoriesandsetproperownershiptoourdeploymentuserandgroup.Notethatyoudonotneedtocreatetheapplicationrootdirectoryifitalreadyexists,andyoudonotneedtosetspecialownershiporpermissionsontherootdirectory.Becauseweareleveragingtherecursivepropertyofthedirectoryresource,therootapplicationdirectorywillbeimplicitlycreated;however,weareconstructingithereforthesakeofcompleteness.

Itiscriticalthatourdirectorieshavethecorrectownershipandpermissions;withoutthis,theapplicationwillbeunabletointeractwiththosedirectoriestostorelogdataorread-and-writeanyconfigurationdata.Thefollowingcodeconstructsthesedirectoriesforusandchangestheownershipandpermissions:

directory"#{app_root}"do

ownernode[:webapp][:user]

groupnode[:webapp][:group]

mode"0755"

action:create

recursivetrue

end

#Createdirectories

["src","logs","conf"].eachdo|child_dir|

directory"#{app_root}/#{child_dir}"do

ownernode[:webapp][:user]

groupnode[:webapp][:group]

mode"0755"

action:create

recursivetrue

end

end

Onethingtonotehereisthatweareusingalooptoconstructourdirectories.Thisisagoodwaytomanagemultipleresourcesofthesametypethathavethesamesetofconfigurationparameters.Herewearesayingthatwehavethreesubdirectories,src,log,andconf.Also,wewanttoconstructadirectoryresourceinsideofourapplication’srootdirectoryforeachsubdirectorywithproperownershipandpermissions.Therecursiveflagissimilartothe-poptiononmkdir,whichtellsittocreateanydirectoriesthataremissinginbetweentherootandthedirectorybeingcreated.

ConstructingyourPythonvirtualenvironmentThismaybenewtonon-Pythondevelopersbutshouldbefairlystraightforward.AvirtualenvironmentoperatesinasimilarwaytoRVMorrbenvforRuby,oraself-containedJARfileforJava.Inthat,itisolatesthePythoninterpreterandinstalledlibrariestoaspecificlocationonthesystem.Inourcase,wewillusethefollowingcodetoachievethis:

python_virtualenv"#{virtualenv}"do

ownernode[:webapp][:user]

groupnode[:webapp][:group]

action:create

interpreter"#{python_interpreter}"

end

Thispython_virtualenvresourcecomesfromthepythoncookbookandwillconstructavirtualenvironmentinthelocationnamedbytheresource(inourcase,thedirectorystoredinvirtualenv,whichaswesawpreviously,isdefinedasthoughinapythondirectoryinsideourapplicationroot)usingthespecifiedinterpreterandownershipproperties.

Avirtualenvironmentwillbecreated,whichcontainsaminimalinstallationofthePythoninterpreteraswellasanyPythonlibrariesthatareinstalledintothevirtualenvironment.Thinkofitasyourapplication’sowninstallationofPythonthatisunaffectedby,andsubsequentlydoesnotaffect,anyotherPythoninstallationonthesystem.

ThisisaveryusefultechniquetoinstallandmanagePythonapplications,andthesameconceptcanbeextendedtotheRailsapplicationusinganysimilartechnologyfromtheRubyworldsuchasRVMorrbenv,asmentionedearlier.

CheckingthesourcecodeOneinterestingthinginthisrecipe,whichhasbeenincludedforfuturereference,istheusageofacookbook,ssh_known_hosts,thatgrabsahost’sSSHkeyandaddsittothesystem’slistofknownSSHkeys.ThisisextremelyusefultodeploysoftwareviaGitHuborBitBucket,whereyouareusingSSHtopulldownthesourcecode,especiallyastheirhostkeysmightchange:

#NeedtoinstallSSHkeyforgithub.com

ssh_known_hosts_entry'github.com'

Notethatitisalsosomewhatinsecureasyouareblindlyacceptingthehost’sfingerprint—ifyouareconcernedaboutsecurity,youcanprovidetheknownfingerprintsmanuallyusingthe:keyattribute.Supplyingthefingerprintisdonethroughthefollowingcode:

ssh_known_hosts_entry'github.com'do

key'github.comssh-rsaAAAAB3NzaC1yc….'

end

Iftherearealargenumberofhostfingerprintsthatyouneedtomanage,oriftheychangefrequently,youcanuseadatabagtostorethem.Ifyouareinterested,lookattheREADMEforthessh_known_hostscookbookformoreexamples.

OncetheSSHkeysareregistered,youcannowclonethesourcefromagit+sshURLsuchasGitHub’sauthenticatedSSHendpoint.

Inthisexample,weareusingapubliclyavailableHTTPSsourcecoderepository;ifyouweretoreplacethiswithyourownSSH-enabledrepository,youwouldneedtochangetherepositoryattributeandalsomakesuretostoreyourdeploymentkeyontheendhost:

#Cloneourapplicationsource

git"#{src_dir}"do

repository"https://github.com/johnewart/simplewebpy_app.git"

action:sync

branch'master'

usernode[:webapp][:user]

end

Byusingthegitresource,therepositorywillbeclonedintothedesignatedsourcedirectoryontheendhost.Here,wewillalsobepullingdatafromthemasterbranchandperformingthisactionasourwebappuser.

InstallinganyextradependenciesTherearetwowaystomodeldependenciesforyourapplication:inyourcookbookandrecipe,orthroughanexternalmechanismsuchasBundler,pip,orotherdependencyresolution,andthedownloadingtooldependingonthelanguageofyourchoice.Aswitheverything,therearebothinherentdrawbacksandbenefitstoeachofthesemethods.

ManagingdependenciesinChefBymodelingyourdependenciesinChef,youhaveaconsistentmodelthatyoucanlooktoinacentralizedlocation.ThismeansthatyourapplicationneedsanewRubygem,oraPythonlibrarythatsomeonemustupdateacookbookorChefconfigurationwiththatinformationinorderforthedeploymenttobesuccessful.Thiscanlimityourabilitytocontinuouslydeploybasedsolelyonthecontentsofasourcecoderepository.Ineffect,thisrequiresyoutomodelthefollowinginChef:

DependentlibrariesLibraryversionsPossibly,thedependenciesofanydeclareddependencies(whichcanspiralquickly)

However,modelingitthiswaydoesensurethatChefhasanaccuratepictureofalltheinformationassociatedwithyourapplication.Thissolutiondoesoffersomeotherbenefits:

DependenciesarepreciselymodeledinChefandcanbequeriedbyothertoolsAnysystem-specificpackagesthatareneededforyourinterpretedlibrariesaregoingtobemodeledbyChefanyway,soit’sallinoneplace(examplescanincludenativeXMLordatabaselibraries)Developerscan’tarbitrarilychangedependenciesandaccidentallybreakdeploymentsbecausetheunderlyinglibrarieshavenotbeeninstalledinproduction

Let’slookatsomethingstothinkaboutwhenusingtoolsexternaltoChefforthistask.

ManagingdependencieselsewhereUsinganexternaltoolsuchasBundlerorpiphassomeadvantages,includingflexibilityandeaseofusebydeveloperswhomaynotbeinvolvedininfrastructureconfiguration.Italsointroducesthepossibilityofmisconfigureddependenciesandunderlyinglibraries.Theprimaryadvantageofthismechanismisthatitprovidesasimplerdependencymanagementmodelfordevelopers—simplyaddarequirementtotheGemfile,requirements.txt,orothermetadatafile,andChefwillautomaticallyinstallthemduringthenextrun.Youalsonowhavetolookintwodifferentplacestodeterminewhatisbeinginstalledonendhosts.Thisalsomeansthatyouarenowconfiguringdependenciesinmultipleplaces,increasingthepossibilityofmakingawrongconfigurationchangeinoneplace.

It’simportanttotakeawaythatthereisnotalwaysonlyonetoolforthejob,anddependingonhowyourorganizationorteamoperates,youmaychoosetomixandmatchhowyoumodeltheapplication-leveldependencies.Forthesakeofdemonstratingthembothtoyou,theapplicationcookbookmodelsthedependenciesintherecipeaswellas

througharequirements.txtfileusingpip.Additionally,youmayfindthatinitiallyyourteamusesonewayandthenmovestoanotherasyourrequirementsstopchangingsofrequently,oryouareabletocombinethemtoyouradvantage.

UsingPython’srequirementsfileOurwebappcookbookhasacustompip_requirementsdefinitionthatprovidesaneasywaytoinstallanyrequirementsstoredinsidearequirements.txtfileintoaspecifiedvirtualenvironmentusingthecopyofpipprovidedbythatvirtualenvironment.Inthefollowingcode,youwillseehowwecanachievethis:

pip_requirements"webapp"do

action:run

pip"#{virtualenv}/bin/pip"

usernode[:webapp][:user]

groupnode[:webapp][:group]

requirements_file"#{src_dir}/requirements.txt"

end

Inthisexample,wearetellingpiptorunasourapplication’suserandgroupandtoinstallthedependenciesinourrequirements.txtintothevirtualenvironmentspecifiedbyvirtualenv.Again,asimilarresourcecanbecreated(ifonedoesnotalreadyexist)toexecuteBundlerforRuby,CPANforPerl,orPEARtomanagePHPdependencies.

ConfiguringyourapplicationNowthatyouhavepreparedyoursystemforyourapplication,youneedtoconfigureit.Inorderfortheapplicationtotalktoourdatabase,youmustprovidetherequireddatabaseconnectioninformationthatwehavestoredinChef.Here,wewilluseatemplatethatisstoredintemplates/default/config.py.erb,andinjectitwithourdatabaseconfiguration.Theresourceforthislookslikethefollowingcode:

template"#{src_dir}/config.py"do

source"config.py.erb"

usernode[:webapp][:user]

groupnode[:webapp][:group]

mode"0600"

variables({

:dbname=>node[:webapp][:dbname],

:dbuser=>node[:webapp][:dbuser],

:dbpass=>node[:webapp][:dbpass],

:dbhost=>dbhost

})

end

Here,weloadourdatabaseinformationontoourtemplateandstoreitinourapplication’sinstalldirectory(wherewecheckedoutthesourceforsimplicity),andsetsomesanefilepermissions.WerethisaRailsapplication,wecanuseasimilartemplatetogeneratedatabase.ymlandmatchingsettings.yml,orifitwereaDropwizardapplication,aservice.ymlfile,aPHPINIfile,oranyothertypeofconfigurationdatathatwereneeded.Inourcase,wearesimplypopulatingthefollowingPythoncodesothatwehaveadatabaseconnectionobject:

importweb

db_params={

'dbn':'postgres',

'db':'<%=@dbname%>',

'user':'<%=@dbuser%>',

'pw':'<%=@dbpass%>',

'host':'<%=@dbhost%>'

}

DB=web.database(**db_params)

cache=False

Thepreviousexampleusestheweb.pydatabasemoduletoconstructanewdatabaseconnectionusingthehash,whichcanthenbeimportedandusedintheotherportionsoftheapplication.Again,thisisagoodstartingexampleforourweb.pyapplicationthatcanbeusedasamodelforwhateverframeworkorapplicationserveryouareusinginyoursystems.

KeepingyourapplicationrunningAllapplicationsneedtobestartedandkeptrunninginsomemanner.IfyouareusingRailswithmod_passenger,thentheApachedaemonwillbetheprimaryentrypointforyourapplication,andthissoftwarewillneedtobeinstalledandconfigured.Inourexample,wewillbeusingthesupervisordservicefromhttp://supervisord.org,whichiswritteninPythonandservesasaveryconfigurable,lightweight,andreliableprocessmanager.Youcanconfigureanentryinthesupervisordsystemconfigurationusingasupervisor_serviceresourcethatisprovidedbythesupervisorcookbookinstalledearlier:

supervisor_service"webapp"do

action[:enable,:restart]

autostarttrue

usernode[:webapp][:user]

command"#{virtual_python}#{src_dir}/server.py#

{node[:webapp][:port]}"

stdout_logfile"#{app_root}/logs/webapp-stdout.log"

stdout_logfile_backups5

stdout_logfile_maxbytes"50MB"

stdout_capture_maxbytes"0"

stderr_logfile"#{app_root}/logs/webapp-stderr.log"

stderr_logfile_backups5

stderr_logfile_maxbytes"50MB"

stderr_capture_maxbytes"0"

directorysrc_dir

environmentenvironment_hash

end

Thepreviousexamplewillgenerateaconfigurationfileforsupervisordwiththesettingsspecifiedinourresourceblock.Unlessyouchangethelocation,theconfigurationfileswillbelocatedin/etc/supervisor.d/[service_name].conf.Inourcase,theserviceisnamedwebapp,anditsconfigurationfilewillbe/etc/supervisor.d/webapp.conf.

Here,wearetellingsupervisordthatwewanttoenableourserviceandthenrestartit(whichwillstartitifit’snotcurrentlyrunning),wherewewanttologtheprocess’soutput,howwewanttorotatethoselogfiles,wheretostartourprocess,whatenvironmentvariablestouse,andmostimportantlywhatcommandtoexecute.

Nowthatwe’velookedatourrecipes,let’sgoaheadandsetupourroles,provisionsomesystems,anddeployourapplication!

DefiningrolesHerewewillconstructourthreeroles,oneeachforourbaseserverconfiguration,databaseserver,andwebserver.Eachrolewillhaveasetofrecipestorun,withthebaseserverprovidingtheuseraccounts,SSHkeys,andothercommoncomponents,andthentheothersprovidingconfigurationdataforPostgreSQLandnginx,respectively.

CreatingthebaseserverroleThekeybitsthatareofinterestinourbaseserverrolearetherunlistandtheconfigurationdatathatspecifywhichgrouptopopulateusersfor.Ifyoulookatthefileroles/base_server.json,youwillseethatwehavedefinedonegroupofuserstopullfromourdatabags:

"override_attributes":{

"shell_users":{

"group":"webapp"

}

}

Andthen,therecipewewanttousethatwillpopulatetheusersonthehostisintherunlist:

"run_list":[

"users::shell_users"

],

InordertoloadtheroleintoChef,youcanissueafromfilecommand:

kniferolefromfilebase_server.json

UpdatedRolebase_server!

Youcanverifythattherolewascreatedwithasimplerolelistcommand:

[user]%kniferolelist

base_server

CreatingthedatabaseserverroleLet’stakealookatsomeportionsofourdatabaseserverrole,asdefinedinroles/postgresql_server.json.ThisfilecontainsthedescriptionofourPostgreSQLserverasmodeledearlierinthechapter.Whatisofinterestinthisfileisouroverride_attributessection;thesearesettingswewanttouseinplaceofthedefaultvaluesprovidedbyourpostgresqlcookbook.Asmentionedbefore,youwillwanttolookatthedocumentationandthedefaultattributes.rbfiletofindoutwhatattributesyoucansetforagivencookbookanditsrecipes.

ThePostgreSQLrecipesuseapostgresqlconfigurationsectionthatcontainsaconfigsectionforserver-specificconfigurationandproperties,andpg_hbafortheauthenticationdata.Lookingatthepostgresqlsection,wecanseethatwewanttoinstallVersion9.3,andwewantittolistenonalladdresses(0.0.0.0)onport5432:

"version":"9.3",

"config":{

"listen_addresses":"0.0.0.0",

"port":"5432"

},

Inaddition,thepg_hbasectioncontainsanarrayofJSONobjectsthatdescribeswhichusershaveaccesstotheservice,bywhatmechanismtheyareabletoauthenticatethemselves(MD5,trusted,localidentservice,andsoon),andfromwheretheycanconnect.Thisiscodedintoourconfiguration,buttherecipescanbeextendedtousedatabagstodeterminethisinformationaswell.Itistoolongtoincludeallofithere,butifyoulookatthepg_hbadata,youwillseethattherearethreeentries:onefortheuserwebusertoconnectfromanywhereusinganMD5password,andtwoforlocaluserstoaccessthedefaulttemplatedatabase,andforthepostgresuseritselftomodifythewebappdatabase.

Inadditiontotheconfigurationdata,thereisarunlist—thistellsChefwhatrecipesthisrolewillinclude.YoucanseefromthisexamplethatwewillbeinstallingthePostgreSQLserverandthenprovisioningourwebapp-specificuseranddatabase(foundincookbooks/pythonwebapp/recipes/database.rb):

"run_list":[

"postgresql::server",

"pythonwebapp::database"

],

Inordertousethis,wewanttoloadourJSONfileintoCheftodefineourdatabaseserverrole:

kniferolefromfilepostgresql_server.json

UpdatedRolepostgresql_server!

Youcancheckthattherolewascreatedwitharolelistcommand:

[user]%kniferolelist

base_server

postgresql_server

CreatingthewebserverroleOurwebapplicationroleislocatedintheroles/web_server.jsonfileandcontainstherequiredinformationforourwebserver.IfyoutakealookattheJSONfile,youwillseethattherunlistcontainsfourentries:

"run_list":[

"python",

"python::virtualenv",

"supervisor",

"pythonwebapp::webapp"

],

BecauseourapplicationreliesonPython,wewanttoinstalltherequiredversionofPythononourhostsaswellasbuildaPythonvirtualenvtoolforourapplication.Inaddition,wewillbeusingsupervisordastheprocessmonitorthatisresponsibleforensuringthatourwebservicestartsandstaysrunning.Wealsoneedtoinstallourwebapplicationoncewehavemetourprerequisites.

SimilartohowweloadedthePostgreSQLrolefromourJSONfile,wecanrepeattheprocessforourwebserverrole:

kniferolefromfileweb_server.json

UpdatedRoleweb_server!

Again,youcancheckthattherolewascreatedwithasimplerolelistcommand:

[user]%kniferolelist

base_server

postgresql_server

web_server

AddingusersWewillneedarecipetomanageourusers;here,wewillusetheuserscookbook.Wewillcreateoneuser,webuser,whichwillbetheaccountthatisusedfordeploymentanduserconnectivity.WewilldefineouruserinaJSONfilesimilartowhatwedidinthepreviouschapter;placethefollowinginausers/webuser.jsonfile:

{

"id":"webuser",

"uid":"1000",

"gid":1000,

"shell":"/bin/bash",

"comment":"Webappdeploymentuser",

"groups":["webusers"]

}

Then,youcanloadthisuserusingthefromfilecommand:

knifedatabagfromfileusersusers

Ensurethatyourhostshaveournewusersbyeditingthebase_serverroleandaddingourwebusersgroupsothatanyusersinthatgroupwillbeprovisionedonallourserversthatincorporatethebase_serverrole:

{

"shell_users":{

"group":"webusers",

}

}

ProvisioningEC2instancesHere,wewillbeprovisioninginstancesinus-west-1,butdependingonwhereyouhaveyourAWSinstancessetup,youwillneedtochangeyourknife.rbconfigurationtospecifytheregionofyourchoice.

Inorderforthemtocommunicatesecurely,wewillconstructasecuritygroupsothatalltrafficbetweenthemispermitted.Thisisoutsidethescopeofthisbook,butitwouldbesomethingtomakesureyouconfigureforproductionsystems,asyouprobablydonotwantthepublicontheInternettohavedirectaccesstoyourdatabaseserver.

Here,wewillassumethatyouhaveyourAWScredentialsandothercriticalcomponentsconfigured,aswecoveredinpreviouschapters.

Toprovisionourdatabaseserver,wewillusethefollowingcommand:

knifeec2servercreate-dubuntu14.04-Iami-ee4f77ab-fm1.small-Zus-

west-1a-Sjewartec2-Ndb00--ssh-userubuntu

Andtoprovisionthewebserver,wewillusethefollowingcommand:

knifeec2servercreate-dubuntu14.04-Iami-ee4f77ab-fm1.small-Zus-

west-1a-Sjewartec2-Nweb00--ssh-userubuntu

Onceyourinstancesareupandrunning,youcannowmoveontoconfiguringthemwiththerolesandconfigurationdatarequired!

ConfiguringthedatabasehostInordertoapplythePostgreSQLroletoourdatabasehost,weneedtomakesureit’sintherunlist.Wecanaccomplishthiswiththefollowingcommand:

knifenoderun_listadddb00"role[base_server]"

knifenoderun_listadddb00"role[postgresql_server]"

Afterensuringthatyournodehasthebase_serverandpostgresql_serverrolesaddedtotherunlist,youcanrunchef-clientonthenewlycreatedhost:

[jewart]%knifessh'name:db00'-xubuntu'sudochef-client'

Oncethisiscomplete,assumingthateverythingwentwell,yournewEC2instancewillhave:

PostgreSQL9.3serverinstalledandrunningAnewdatabase(thenameofwhichisdefinedfromyourconfiguration)AdatabaseuserthatisgrantedpermissiontoconnectCorrectpg_hba.confandpostgresql.conffilesforourservice

Nowthatwehaveourdatabaseserverconfigured,whichcanbeverifiedbyloggingontotheserverandensuringthattheserviceisrunning,let’stakealookatsettingupourwebapplication.

ConfiguringthewebserverInorderforthewebservertodeploythewebapp,weneedtoaddtherequiredrolestothewebserver,aswedidwiththedatabaseserver:

knifenoderun_listaddweb00"role[base_server]"

knifenoderun_listaddweb00"role[web_server]"

Now,wecanexecutechef-clientonthewebhost(againmakingsuretousesudosothatithaspermissiontodoitswork):

[jewart]%knifessh'name:web00'-xubuntu'sudochef-client'

Atthispoint,ourwebserverwillbeinthefollowingstate:

Thefollowingrequiredpackageswillbeinstalled:

Python2.7anddevelopmentlibrariesThePostgreSQLclientdevelopmentlibrariesGit

ThedirectoriesourapplicationneedstorunarecreatedAvirtualenvtool,whichisbasedonthesystemPython2.7iscreatedOurapplicationhasbeencheckedoutfromGitHubAconfigurationfilein/opt/webapp/src/config.pyiscreatedbyChefSupervisordisconfiguredtorunourapplicationandstartstheserver.pydaemon

Now,youshouldbeabletovisityournewlyinstalledwebapplicationatthefollowingURL:http://your-new-ec2-instance-hostname:8080

Ifyoudon’tseeyourapplication,makesurethateachoftheprecedingstepswassuccessful.

DeployingyoursoftwareDeployingsoftwareshouldbetreatedjustlikeyourinfrastructure;repeateddeploymentsofthesamecommitandthesameconfigurationshouldyieldaconsistentstateofyourenvironment.Inthisexample,wewillbeabletodeploynewupdatestoourwebapplicationsimplybyupdatinganynodesthatusetheweb_serverrole.ThecombinationofourrecipeandourconfigurationdatawiththesourcecodehostedinourGitHubrepositorywillensurethatthemostup-to-dateconfigurationandsourceareplacedonourhost.

ManuallydeployingupdatesFuturedeploymentsonlyrequirepushingchangestothemasterbranchandthenrunningchef-clientonanywebserversthatareinthefleet.Thiscanbeaccomplishedonasinglehost(web00)usingthefollowingcommand:

knifessh'name:web00'-xubuntu'sudochef-client'

ThiswilltellknifethatwewanttoSSHtothehostwhosenameisweb00astheubuntuuser(becausethat’sthedefaultEC2userwithsudoaccess)andexecutechef-clientasrootviasudo.Thiswillworkwellifweonlyhaveonehost;however,asyourcapacityincreasesandyouhavemultiplehosts,youwilllikelywanttoexecutethisonagroupofhostsinthefuture.ThiscanbeaccomplishedusingthesearchcapabilityofChefthatallowsyoutoexpandalistofnodesthatmatchasetofcriteria.Here,wewillwanttobuildalistofallthehoststhathaveourweb_serverroleassociatedwiththem.Thefollowingcommandwillaccomplishthis:

knifessh'role:web_server'-xubuntu'sudochef-client'

ThiswilluseChef’ssearchtofindallnodeswiththeweb_serverroleandthenSSHtothemsequentially,thesameasbeforebutonlyacrossmultiplehostsinsteadofjustone.

AutomatingdeploymentThewebapplicationrecipeisdesignedsothatitsyncsthesourcewiththeupstreamGitHubrepository.Bydoingthis,wecanexecutetherecipemultipletimes,andanytimethereareupdates,theywillbepulleddownontothelocalhost.Ifwewantedto,theprocessofdeploymentscouldbeautomatedinthefollowingfashion:

Activedevelopmentoftheapplicationhappensinaseparate,developmentbranchCodeistestedthoroughlyandthenmergedintoamaster(orwhateverbranchisbeingdeployedontohosts)whenitisstableandreadyforproductionHostsareconfiguredtorunchef-clientonafixedintervalusingatoolsuchascronandwillautomaticallyupdatethemselves

Thepossibleissueswiththisarethatbadcodegetsautomaticallydeployedtoendhostsandsoon.However,withenoughintegrationtestingandahighenoughconfidencelevel,ourcodeshouldbesafetodeploytoproductionifitisinthemasterbranch.Throughacombinationoftagsandpropersourcemanagement,rollbackscouldbeassimpleasrevertingthedeploybranchtoaknown-goodtagandtheywouldhappenassoonasthenextchef-clientexecutionorforcedusingknifeasoutlinedpreviously.

SummaryAmajorattractiontoChefandinfrastructureautomationistheabilitytodeploysoftwareandprovisionsystemsquicklyandconsistently.Usingthecookbooksandexamplesoutlinedinthischapter,youshouldbeabletomodelyourapplicationanditscomponents,gathercookbooksrequiredtodeployneededsoftware,buildcookbookstoconfigureanddeploycustomsoftware,andextendtheexamplestoprovidemorefunctionalitiesorenhanceyourinfrastructure.

Nowthatyouhaveseenhowtotakeanapplicationfromdevelopmenttodeployment,let’stakealookatsomemoreadvancedexamplesofcookbookdevelopment,includingwritingcustomprovidersandresources,workingwithsecuredata,searchingChef,andotherwaysofenhancingourrecipesandcookbooks.

Chapter7.BeyondBasicRecipesandCookbooksSofar,wehaveonlyreallylookedathowtousecookbooksasaconsumer,notasaproducer.InordertoharnessthetruepowerofChef,itisimportanttolearnhowtobuildourowncookbooksandrecipesusingthefullfeaturesetthatChefprovides.Thischaptercovers:

AdvancedrecipeconceptsManagingyourdatausingdatabagsSearchingCheffromrecipesAdvancedscriptinginrecipesAuthoringcustomproviders,resources,anddefinitionsDealingwithencrypteddata

ManagingusersBasicusermanagementinChefisachievedthroughtheuseoftheuserresource.Thisresourceallowsyoutoadd,remove,orotherwisemanipulateusersonyourhosts.However,youcan’tpossiblywriterecipesthatcontainoneresourceperuser;itsimplywouldn’tscale.Inordertomakelarge-scaleusermanagementeasier,wecancombinesomeofChef’scapabilitiessuchasdatabags,per-role,per-node,andper-environmentconfigurationtoenablescalableusermanagement.

Let’stakealookatausercookbookthatcanprovidetheseabilities.

EvolutionofashelluserrecipeFirst,let’stakealookataverynaiveusermanagementrecipe.Thiscookbookhasahardcodeduserslist;initially,itcontainsfrodoandsamwiseandsimplyiteratesthroughthelist,creatingusersasitgoes.Hereiswhatthelistmaylooklike:

users=[

{

'id'=>'frodo',

'uid'=>'100',

'gid'=>100,

'shell'=>'/bin/hobbitshell',

'comment'=>'Frodooftheninefingers'

},{

'id'=>'samwise',

'uid'=>'101',

'gid'=>101,

'shell'=>'/bin/gardenshell',

'comment'=>'Samwisethestrong'

}

]

users.eachdo|u|

home_dir="/home/#{u['id']}"

useru['id']do

uidu['uid']

gidu['gid']

shellu['shell']

commentu['comment']

supports:manage_home=>true

homehome_dir

end

end

Thisapproachwillworkforahandfulofusers,butithastheproblemofbeingverylimitedinscopeanddifficulttomaintain.Italsoisolatesthelistofuserstothisrecipe,makingitdifficulttoaccessdatafromotherrecipesandverybrittle.Thefirstthingwecanimproveismaketheusersaccessibletothisandanyotherrecipethroughtheuseofdatabags.Let’stakealookathowwecanusedatabagstomakeuserdatamanagementsimplerandmoreflexible.

StoringdataindatabagsDatabagsaredesignedtostorearbitraryconfigurationdatathatpertainstoyourentireinfrastructure.Thismayincludeusers,globalsettings,firewallrules,andsoon;ifitcanbemodeledusingbasicJSONdatastructuressuchasarraysanddictionaries,itcanbeincludedinadatabag.Wehaven’ttouchedmuchontheseyet,sonowisagoodtimetotakealookatwhattheycandowhilemodelingusers.

CreatingadatabagforusersDatabagsarecollectionsofdatathatarerelatedtooneanother;forexample,users,firewallrules,databaseservers,andsoon.Herewewillcreateadatabagthatcontainsouruserdata.ThisisnotintendedtobeareplacementforadirectoryservicesuchasLDAP,thoughyoucouldpotentiallyuseittostoreallyouruserdataandthenwriterecipestopopulateanLDAPserverwithuserdata(inthisway,youmaybeabletokeepanActiveDirectorysystemandaseparateLDAPsysteminsyncbymakingyourChefdatabagtheauthoritativesourceforuserdata).Let’stakealookathowtocreateandmanipulateadatabagwithuserinformation:

[jewart]%knifedatabagcreateusers

Createddata_bag[users]

Now,createanewuser,frodo(youwillneedtohavetheEDITORvariablesettoatexteditorsuchasvimonLinuxsystems):

[jewart]%exportEDITOR=vim

[jewart]%knifedatabagcreateusersfrodo

Databagusersalreadyexists

Youwillbepresentedwithanewentitytemplatethatcontainsonlyonekey,id,whichissettothenameoftheentityyoucreated;inourcase,frodo:

1{

2"id":"frodo",

3}

Savethisfileandyouwillnowhaveone,mostlyempty,entityinyourusersdatabagnamedfrodo.Youcancheckthiswiththeshowsubcommand:

[jewart]%knifedatabagshowusers

frodo

Everyiteminadatabaghastohaveauniqueidentifier,whichcanbemeaningfulorjustarandomidentifier;inourcase,itwilldoubleupastheloginnamefortheuser.WecantakeourpreviousdatafromtherecipeandconvertthattodatabagelementsbywritingthemtoJSONfilesanduploadingthemwithknife.Totakeadvantageofuploads,wecancreateadirectory,users,andcreateoneJSONfileperentry:

{

"id":"frodo",

"uid":"100",

"gid":100,

"shell":"/bin/hobbitshell",

"comment":"Frodooftheninefingers"

}

{

"id":"samwise",

"uid":"101",

"gid":101,

"shell":"/bin/gardenshell",

"comment":"Samwisethestrong"

}

Onceyouhavecreatedthese,youshouldhavetwofiles,frodo.jsonandsamwise.jsoninsideausersdirectory.Inordertobulkuploadthem,weuseaknifedatabagfromthe<dir><databagname>fileinthefollowingmanner:

[jewart]%knifedatabagfromfileusersusers

Updateddata_bag_item[users::frodo]

Updateddata_bag_item[users::samwise]

Youcanverifywhethertheentrieswerecreatedcorrectlywiththeknifedatabagshow<databag><entity_id>command:

[jewart]%knifedatabagshowusersfrodo

comment:Frodooftheninefingers

gid:100

id:frodo

shell:/bin/hobbitshell

uid:100

SearchingfordataNowthatwehaveourdatainadatabaginChef,wecansearchforitusingthesearchcriteria.Forexample,ifwewantedonlyalluserswhosenamesstartwiththeletters,wecansearchwiththefollowingcommand:

[jewart]%knifesearchusers'id:s*'

1itemsfound

chef_type:data_bag_item

comment:Samwisethestrong

data_bag:users

gid:101

id:samwise

shell:/bin/gardenshell

uid:101

Alternatively,ifwewantedalltheusersinagivendatabag,wecanperformthefollowingsearch:

[jewart]%knifesearchusers'id:*'

2itemsfound

chef_type:data_bag_item

comment:Frodooftheninefingers

data_bag:users

gid:100

id:frodo

shell:/bin/hobbitshell

uid:100

chef_type:data_bag_item

comment:Samwisethestrong

data_bag:users

gid:101

id:samwise

shell:/bin/gardenshell

uid:101

SearchinginsiderecipesNowthatwehavesomedatabagdatacreatedandcanperformbasicsearches,let’sseehowwecanusethattoenhanceourrecipeusingthebuilt-insearchmethod.Thisallowsustoperformthesearcheswejustranwithknifeinsideourrecipes.Thesearchmethodhasasimilarformattotheknifecommand:

search(search_scope,search_criteria)

Thefollowingaresomesimpleexamples:

all_users=search(:users,'id:*')

users_s=search(:users,'id:s*')

all_nodes=search(:node,'*')

Withthis,wecanenhanceourshelluserrecipetousetheentitiesintheusersdatabagratherthanhardcodethem.Ournewrecipewouldlooklikethefollowing:

#Replacethehard-codedusersarraywithasearch:

users=search(:users,'id:*')

#Sameasbefore,we'vejustmovedourdatasource

users.eachdo|u|

home_dir="/home/#{u['id']}"

useru['id']do

uidu['uid']

gidu['gid']

shellu['shell']

commentu['comment']

supports:manage_home=>true

homehome_dir

end

end

Thisisjustasimplesearch;thiswillworkforasmall-scaleinfrastructurewithafixedsetofusers,wherethere’snoneedtorestrictcertaingroupsofuserstocertainhosts.Youcaneasilyimagine,however,asituationwheresomeusersareprovisionedonlytocertainhoststhroughgroups.Let’slookathowwecanachievethiswithsomebetterusermetadataandamoreadvancedsearch.

EnhancingyourusercookbookInourpreviousexample,weusedthesearchmethodtofindalloftheusersinouruser’sdatabag.Herewewillgoonestepfurthertoisolateusersbasedonarbitrarygroupsandseehowwecanlimitthelistofuserstobeprovisionedusingacombinationofsearch,usermetadata,andnodeconfiguration.

First,weneedtoaddagroupskeytoourusers.Let’saddthattoourexistinguserJSONdatafilesandaddafewmoreusers,legolasandgimli:

{

"id":"frodo",

"uid":100,

"gid":100,

"shell":"/bin/hobbitshell",

"comment":"Frodooftheninefingers",

"groups":["hobbits","fellowship"]

}

{

"id":"gimli",

"uid":201,

"gid":201,

"shell":"/bin/csh",

"comment":"Grumpyolddwarf",

"groups":["dwarves","fellowship"]

}

{

"id":"legolas",

"uid":200,

"gid":200,

"shell":"/bin/zsh",

"comment":"KeeneyedLegolas",

"groups":["elves","fellowship"]

}

{

"id":"samwise",

"uid":"101",

"gid":101,

"shell":"/bin/gardenshell",

"comment":"Samwisethestrong",

"groups":["hobbits","fellowship"]

}

Onceagain,weupdatetheexistingrecordsandcreateournewrecordsusingknifedatabagfromfile:

[jewart]%knifedatabagfromfileusersusers

Updateddata_bag_item[users::frodo]

Updateddata_bag_item[users::gimli]

Updateddata_bag_item[users::legolas]

Updateddata_bag_item[users::samwise]

Nowthatyouhaveafewadditionalusersinyourdatabag,andeachuserhassomegroupmetadataattachedtoit,let’stakealookathowwecanusethistoprovisiononlycertainusersonspecifichosts.First,weneedtobeabletolimitoursearchscopedynamically;otherwise,wewillneedtomodifyourrecipeonaper-hostbasisandthatjustwon’tscale.Weneedtoaddadynamicsearchquerytoourrecipewithsomethinglikethefollowingcode:

search_criteria="groups:#{node[:shell_users][:group]}"

Thiscreatesasearchcriteriastringthatwillmatchobjectsthathavethevaluespecifiedsomewhereintheirgroupskey.Inordertomakethisdynamicperhost,wewillstorethisvalueinashell_usershashunderthegroupkey.Forexample,ifyouwantedtoaddallusersthatareinthehobbitsgrouptoaspecificnode,thenyournode’sconfigurationwouldneedtocontainthefollowing:

{

"shell_users":{

"group":"hobbits",

}

}

Thiswillbuildasearchcriteriaof"groups":"hobbits",whichifwepasstothesearchmethodwillyieldallentriesintheusersdatabagthathave"hobbits"insidetheirgroupslist.Considerthefollowingrecipecode:

users=search(:users,search_criteria)

Thenodeconfigurationdatawillexpandthesearchcriteriaduringanexecutiononthisnodetobethefollowing:

search_criteria="groups:hobbits"

Giventhedatawehavestoredinourusersdatabag,thiswouldmatchsamwiseandfrodoastheyhavethehobbitsgroupintheirgroupslist.Wecanverifythisbytryingthesamesearchonthecommandlinewithknife:

[jewart]%knifesearchusers"groups:hobbits"

2itemsfound

chef_type:data_bag_item

comment:Frodooftheninefingers

data_bag:users

gid:100

groups:

hobbits

fellowship

id:frodo

shell:/bin/hobbitshell

uid:100

chef_type:data_bag_item

comment:Samwisethestrong

data_bag:users

gid:101

groups:

hobbits

fellowship

id:samwise

shell:/bin/gardenshell

uid:101

Asyoucansee,thisallowsustonarrowlydefinethelistofuserstobemanagedthroughthecombinationofentitymetadataanddynamicsearchcriteria.Youcanbuildmoreadvancedapplicationsusingthismethodologywithmoreadvancedsearchcriteriaandincorporatingmoreoftheentities’metadata.

DistributingSSHkeysInadditiontomanaginguseraccounts,wecanalsouseCheftomanageSSHkeys.Becauseagivenuser’sacceptedSSHkeysarestoredinaper-userconfigurationfile,itisquitesimpletomanipulatethem.BycreatingatemplateforSSH-authorizedkeys,wecanbuildarecipethatwilltaketheSSHkeydatafromthedatabagandpopulatetheauthorizedkeysfileonthehost.Bydoingthis,users’SSHkeyscanbestoredinChefanddistributedtoanynumberofhostswithjustonecommand.ThissolvestheproblemstypicallyassociatedwithdistributionandrevocationofSSHkeysinsideanorganization.

TemplatingtheauthorizedkeysHereisasampletemplatewewilluseforouruser’sauthorizedkeysfile;thiswouldbedefinedinanauthorized_keys.erbfile:

<%if@ssh_keys.is_a?(Array)%>

<%=@ssh_keys.join("\n")%>

<%else%>

<%=@ssh_keys%>

<%end%>

Thisisaverysimpletemplatethathasonlytwocases:ifthetemplatevariablessh_keysisanarray,itwillprintthemoutwithanewlineinbetweenthem;otherwise,itwillsimplyprintoutthecontentsofthevariable.

Tousethistemplate,wewillsimplyprovideitwithalistofSSH-compatiblekeystrings:

template"#{home_dir}/.ssh/authorized_keys"do

source"authorized_keys.erb"

owneru['id']

groupu['gid']||u['id']

mode"0600"

variables:ssh_keys=>u['ssh_keys']

end

Now,wecanmodifyoneofourprevioususerJSONentitiestoaddSSHkeys:

{

"id":"frodo",

"uid":100,

"gid":100,

"shell":"/bin/hobbitshell",

"comment":"Frodooftheninefingers",

"groups":["hobbits","fellowship"],

"ssh_keys":[

"ssh-dss

RG9uJ3Qgd29ycnksIFNhbS4gUm9zaWUga25vd3MgYW4gaWRpb3Qgd2hlbiBzaGUgc2VlcyBvbmU

ufrodo@shire",

"ssh-dss

TXkgbWFzdGVyLCBTYXVyb24gdGhlIEdyZWF0LCBiaWRzIHRoZWUgd2VsY29tZS4gSXMgdGhlcmU

gYW55IGluIHRoaXMgcm91dCB3aXRoIGF1dGhvcml0eSB0byB0cmVhdCB3aXRoIG1lPyA=

sauron@mordor"

]

}

%knifedatabagfromfileusersssh_keys/frodo.json

Updateddata_bag_item[users::frodo]

Onceyouruserhasbeenupdated,checkwhetheryournewlyaddedmetadatahasbeenupdated,lookingforyournewssh_keyskeyintheentity.Inordertodothat,youcanshowthecontentsofyourdatabagusingthefollowingcommand:

%knifedatabagshowusersfrodo

TheoutputofthisshouldlineupwithyournewlyupdatedJSONcontent.Withtheseadded,wecanwriteanewrecipethatwillallowustodeployauthorized_keysfilesforeachuseronourhosts.OurrecipewillusethesamesearchcriteriafromourpreviousrecipeaswewanttoapplyourSSHkeystoallofourshellusers.

ThisrecipeisresponsibleformakingsurethattheproperdirectoryforSSHiscreatedandhasthecorrectpermissions,aswellascreatingtheauthorized_keysfilewiththenecessarypermissionsandstoringtheSSHkeysassociatedwiththeuserin/home/user/.ssh/authorized_keys:

search_criteria="groups:#{node[:shell_users][:group]}"

search(:users,search_criteria)do|u|

home_dir="/home/#{u['id']}"

directory"#{home_dir}/.ssh"do

owneru['id']

groupu['gid']

mode"0700"

recursivetrue

end

template"#{home_dir}/.ssh/authorized_keys"do

source"authorized_keys.erb"

owneru['id']

groupu['gid']

mode"0600"

variables:ssh_keys=>u['ssh_keys']

end

end

AddingdeploymentkeysIfyouhaveeverdeployedaRailsapplicationtohoststhatneedtohaveaccesstoyoursourcecodeinaGitHuborBitBucketrepository,thenyouwillknowhowhandyitistomanagedeploymentkeysacrossafleetofhosts.Wecaneasilygeneratearecipethatlooksatanode’slistofdeploymentusersfollowingourpreviousexamplesasastartingpoint.Here,welookfordeployusersinsteadofshellusers,asthesearetheoneswewanttomanagedeploymentkeysfor.Notethatinthisexample,theseuserswouldalsoneedtobeincludedintheshell_usersgrouptoensurethattheygetcreatedbyourpreviousrecipe:

search_criteria="groups:#{node[:deploy_users][:group]}"

search(:users,search_criteria)do|u|

home_dir="/home/#{u['id']}"

directory"#{home_dir}/.ssh"do

owneru['id']

groupu['gid']||u['id']

mode"0700"

recursivetrue

end

template"#{home_dir}/.ssh/id_rsa"do

source"deploy_key.erb"

owneru['id']

groupu['gid']||u['id']

mode"0600"

variables:key=>u['deploy_key']

end

end

Tousethisnewrecipe,thedeploymentuserswouldneedtobemodifiedtoincludeagroupidentifierandtheirprivatekey.Thegroupwouldbereservedforusersinvolvedindeployingyourapplicationandbeaddedtotheuser’sgroupskeyinChef.Additionally,anunencryptedSSHprivatekeywouldneedtobepresentinadeploy_keyfield.

TipIncludingunencryptedSSHkeyscanposeasecurityrisk.Thiscanbemitigatedusingencrypteddatabagsoranexternalsecuritymaterialmanagementservice.

WritingcustomextensionsWithChef,youaregivenimmediateaccesstoanumberofresources:files,users,packages,templates,andsoon.However,therewillalwaysbetimeswhenthisdoesnotprovideyouwitheverythingthatyouneed.Fortunately,thebuilt-inChefresourcesorLWRPs(light-weightresourceproviders)arejustRubycodeandwerebuiltwiththeintentionofprovidingaframeworkforenduserstobuildtheirowncomponents.Thismeansthatyoucaneasilybuildyourowncustomresources,andthesecanbesharedwithothersjustlikeanybuilt-inLWRP.

DevelopingacustomdefinitionOneofthesimplestresourcesthatwecanbuildisadefinition—adefinitionislikearesourcewithonlyonebuilt-inprovider.Thesecanbethoughtofasreusablemodulesthatyoucanleverageinsideofyourrecipes.Ifyoufindyourselfwritingthesamethingrepeatedlyinyourrecipes,thenitisprobablyagoodcandidatetowriteacustomdefinition.Forexample,let’slookathowwecanbuildtwodifferentdefinitions:oneforexecutingPython’sPIP,thePythonpackageinstallationtool,toinstallthecontentsofarequirements.txtfileforaPythonapplication,andanothertoinstallapplicationsthatfollowthesamepattern.

OrganizingyourcodeAsdiscussedbefore,cookbookscanhaveadefinitionsdirectoryinsidethem.Thecontentsofthisdirectoryareincludedinyourcookbookrunsandshouldhaveoneperdefinition.ForourPIPresource,wewillcreateafile,definitions/pip_requirements.rb,andforourapplicationtemplate,definitions/python_web_application.rb.Thesewilleachcontaintherespectivedefinitions.

WritingadefinitionforusingPIPDefinitionslooklikeanyChefcomponent—theyarecomposedofresources,variables,scripts,andanythingelseyouuseinarecipe.However,unlikearecipe,theyaredesignedtobereused.Wherearecipeisdesignedwithaspecificeffectinmindsuchasdeployingaspecificapplication,thedefinitionisdesignedtobeconsumedbyrecipestoreduceduplicatecode.

Eachdefinitionisencapsulatedinadefineblock,ano-opversion,orourPIPexamplewouldlooklikethis:

define:pip_requirementsdo

end

Thisexampledoesabsolutelynothing,butitcanbeusedinarecipeasfollows:

pip_requirements"app_requirements"do

end

Justinthesamewayyouwoulduseafile,user,ortemplateblockinyourrecipe,youcanuseyourcustomdefinitions.Now,let’senhanceourdefinitionbyusingthenameparameter—thestringargumentpassedtothepip_requirementsblockinyourrecipe;here,itisapp_requirements:

define:pip_requirements,:action=>:skipdo

name=params[:name]

end

Eachinvocationofadefinitionpassestheparametersintheblocktothedefinition;theseareaccessedinsidethedefinitionthroughtheparamshash.Thereisonespecialparameter,:name,whichcancomefromthefirstargumentbeforetheblock,asshowninthepreviouscode,orfromthenameparameterinsidetheblock.Thisisaconvenienceparameter

designedtomakerecipesmorereadablebyallowingthedevelopertowrite:

resource"somehandydescription"do

...

end

Thiscodeiseasiertoreadthan:

resourcedo

name"somehandydescription"

end

Giventhisinformation,let’slookatthePIPexamplefrompip_requirements.rb:

define:pip_requirements,:action=>:skipdo

name=params[:name]

requirements_file=params[:requirements_file]

pip=params[:pip]

user=params[:user]

group=params[:group]

ifparams[:action]==:run

script"pip_install_#{name}"do

interpreter"bash"

user"#{user}"

group"#{group}"

code<<-EOH

#{pip}install-r#{requirements_file}

EOH

only_if{File.exists?("#{requirements_file}")andFile.exists?("#

{pip}")}

end

end

end

Here,thedefinitionexpectsfivearguments:theresourcename,thepathtotherequirements.txtfile,thepipbinarytouse,aswellastheuserandgrouptoexecutepipas.ThereasonthattheresourceacceptsthepathtopipistoallowusingpipinsideaPythonvirtualenvironment.Bydoingthis,thedefinitionbecomesalittlemoreflexibleinsituationswhereyouneedtoinstallyourrequirementsintoadifferentlocationonthesystem.

Alsonotethatwecandefinedefaultparametersaspartofthedefinition’ssignature:

define:pip_requirements,:action=>:skipdo

Inthiscase,thedefaultactionis:skip,butitcanbesettoanythingyouwantittobe.Hereitissetto:skipsothatitonlygetsinvokeddeliberatelyratherthanbyvirtueofbeingusedinarecipe.

Asthisisasimpledefinition,itonlycontainsoneresource—ascriptblockthatwilleffectivelyexecutepipinstall-r/path/to/requirements.txtasthespecifieduserandgroup.Anexampleuseofthisdefinitioncanbeseenasfollows:

pip_requirements"requirements"do

action:run

pip"/usr/local/bin/pip"

usernode[:app][:user]

groupnode[:app][:group]

requirements_file"#{app_root}/src/requirements.txt"

end

Thiscanbeusedinplaceofthebuilt-inscriptresource:

script"pip_install_#{name}"do

interpreter"bash"

usernode[:app][:user]

groupnode[:app][:group]

code<<-EOH

/usr/local/bin/pipinstall-r#{app_root}/src/requirements.txt

EOH

only_if{

File.exists?("#{app_root}/src/requirements.txt")and

File.exists?("/usr/local/bin/pip")

}

end

FollowingChef’sdeclarativelanguage,buildingdefinitionssuchasthisonemakesitmoreobviousastowhatishappening,ratherthanhowitishappening.Wehaveabstractedtheshellscriptandguardtestsbehindafaçade,thatisthepip_requirementsdefinition,whichismoreclearinitseffectwhenyoureadarecipe;youdon’tneedtoexaminethecontentsofthescriptblocktodeducewhatishappeningastheresourcenametellsyouexactlywhat’sgoingtobedone.

DefiningafullapplicationtemplateIfyouhaveapplicationsthatfollowthesamestructure(thinkapplicationsthatuseacommonframeworksuchasRails,Django,Pyramids,Python-tornado,andsoon),thenyouwouldlikelywanttodefineacommondefinitionforwhatsuchanapplicationlookslike.ConsiderhereadefinitiontoinstallaPythonwebapplicationfromGitHubusingsomecommonidioms:

define:tornado_applicationdo

app_name=params[:name]

app_root=params[:app_root]

app_user=params[:user]

app_group=params[:group]

python_interpreter=params[:python_interpreter]||

"/usr/bin/python3.3"

github_repo=params[:github_repo]

deploy_branch=params[:deploy_branch]||"deploy"

virtualenv="#{app_root}/python"

virtual_python="#{virtualenv}/bin/python"

app_dir="#{app_root}/src/#{app_name}"

#NeedtoinstallSSHkeyforGitHub

#thiscomesfromthessh_known_hostscookbook

ssh_known_hosts_entry'github.com'

#Basepackagerequirements

package"git"

package"libpq-dev"

package"libxml2-dev"

package"python3.3"

package"python3.3-dev"

directory"#{app_root}"do

owner"#{app_user}"

group"#{app_group}"

mode"0755"

action:create

recursivetrue

end

#Createdirectories

["bin","src","logs","conf","tmp"].eachdo|child_dir|

directory"#{app_root}/#{child_dir}"do

owner"#{app_user}"

group"#{app_group}"

mode"0755"

action:create

recursivetrue

end

end

#InstallPythonvirtualenv

python_virtualenv"#{virtualenv}"do

owner"#{app_user}"

group"#{app_group}"

action:create

interpreter"#{python_interpreter}"

end

#Applicationcheckout

git"#{app_dir}"do

repository"#{github_repo}"

action:sync

user"#{app_user}"

branch"#{deploy_branch}"

end

#Pythondependenciesforapp

pip_requirements"tornado_app[#{app_name}]"do

action:run

pip"#{virtualenv}/bin/pip"

user"#{app_user}"

group"#{app_group}"

requirements_file"#{app_dir}/requirements.txt"

end

end

Thisdefinitioncanbeusedasshowninthefollowingexample:

tornado_application"image_resizer"do

app_root"/opt/webapps"

user"webapp"

group"webapp"

deploy_branch"master"

github_repo"git@github.com:myorg/image_resizer.git"

python_interpreter"/usr/bin/python3.3"

end

Accordingtothepreviousdefinition,thiswoulddothefollowing:

Addasystem-wideSSH-knownkeyforgithub.com(requiredtoperformaGitcloneandguaranteesthatfuturekeychangeswillwork)Installanyrequiredpackagesiftheydidn’talreadyexist,includingGit,Python,andthepostgresqlclientEnsureanyapplication-requireddirectoriesexistfordatasuchasbinaries,logs,configuration,andmoreCreateaPythonvirtualenvironmentbasedonthesuppliedPythoninterpreter(3.3)in<app_root>/python

Cloneorsync(ifitwasalreadycloned)thesourcecodefrom<github_repo>to<app_root>/src/<app_name>

Installtherequirementsspecifiedin<app_root>/src/<app_name>/requirements.txtusingthecopyofpipfromthevirtualenvironmentin<app_root>/python

Assumingyouhadanothersimilarlystructuredapplication,butyouwantedtouseadifferentuser,group,Pythoninterpreter,anddeploymentbranch,youcaneasilyconfigureitusingthefollowingresource:

tornado_application"restful_api"do

app_root"/opt/webapps"

user"restapi"

group"restapi"

deploy_branch"production"

github_repo"git@github.com:myorg/restful_api.git"

python_interpreter"/usr/bin/python3.2"

end

Asyoucansee,definitionsallowustodefinereusableresourcesinChef.Therearethreeprimarybenefitstothisapproach:

Simplifiedrecipesareeasiertoread,haveclearerintentions,andlesscodetoaudit,whichmakesthemlesserrorproneAnychangestothedefinitionareautomaticallyappliedtoanyusageofthedefinition,whichmeansyoudon’tneedtomaintainmultiplevariationsIt’seasiertotestbecauseit’sdesignedtobeparameterizedandmodular

NowthatyouseehoweasyitistowritecustomresourcesforChefthroughdefinitions,let’sexaminewritingafull-blownresourcethathasaseparateproviderimplementation.

BuildingaresourceAChefLWRPiscomposedoftwoprimarycomponents,aresourceandaprovider.Theresourceistheblueprintforwhatisbeingprovided;itdescribestheresource,includingwhatactionscanbetakenbyaresource,thepropertiesthatdescribetheresource,andanyotherhigh-levelinformationaboutit.Theproviderisresponsiblefortheactualimplementationoftheresource.Inprogrammingterms,theresourceisanabstractclassorinterfacewheretheproviderisaconcreteclassorimplementation.Forexample,oneofChef’sbuilt-inresourcesisthepackageresource;however,thisisaveryhigh-levelresource.Thepackageresourcedescribeswhatapackageisandwhatapackagecandobutnothowtomanagethem.Thatworkislefttotheproviders,includingRPM,APT,FreeBSDpackages,andotherbackendsystemsthatarecapableofmanagingon-diskinstallationofpackages.

DefiningtheresourceAsanexample,let’stakealookatanS3bucketresource:

actions:sync

default_action:syncifdefined?(default_action)#Chef>10.8

#DefaultactionforChef<=10.8

definitialize(*args)

super

@action=:sync

end

#TargetfolderonthehosttosyncwiththeS3bucket

attribute:destination,:kind_of=>String,

:name_attribute=>true

#Anythingtoskipwhensyncing

attribute:omit,:kind_of=>Array

#AWSAccess/secretkey

attribute:access_key_id,:kind_of=>String

attribute:secret_access_key,:kind_of=>String

Here,ourresourceisanS3bucketthatdeclarestheactionsitcantakealongwiththeattributesthatitrelieson.Here,ourresourcedeclaresthatithasoneavailableaction,sync,whichisthedefaultactionandthatithasfourattributes:thedestination,whatfilestoskip,theaccesskey,andthesecretkey.

ImplementingtheproviderTheprovideriswherethelogicfortheresourceisplaced—itisresponsibleforactingontheresourcebeingdescribed.ForourS3bucket,itlookslikethefollowing:

require'chef/mixin/language'

#Onlyrunasneeded

defwhyrun_supported?

true

end

action:syncdo

Chef::Log.debug("Checking#{new_resource}forchanges")

fetch_from_s3(new_resource.source)do|raw_file|

Chef::Log.debug"copyingremotefilefromorigin#{raw_file.path}to

destination#{new_resource.destination}"

FileUtils.cpraw_file.path,new_resource.destination

end

new_resource.updated_by_last_action(true)

end

defload_current_resource

chef_gem'aws-sdk'do

action:install

end

require'aws/s3'

current_resource=new_resource.destination

current_resource

end

deffetch_from_s3(source)

begin

protocol,bucket=URI.split(source).compact

AWS::S3::Base.establish_connection!(

:access_key_id=>new_resource.access_key_id,

:secret_access_key=>new_resource.secret_access_key

)

bucket.objects.eachdo|obj|

name=obj.key

if!new_resource.skip.contains(name)

Chef::Log.debug("Downloading#{name}fromS3bucket#{bucket}")

obj=AWS::S3::S3Object.findname,bucket

file=Tempfile.new("chef-s3-file")

file.writeobj.value

Chef::Log.debug("File#{name}is#{file.size}bytesondisk")

begin

yieldfile

ensure

file.close

end

else

Chef::Log.debug("Skipping#{name}becauseit'sintheskiplist")

end

end

rescueURI::InvalidURIError

Chef::Log.warn("ExpectedanS3URLbutfound#{source}")

nil

end

end

Let’stakealookattheprovider,piecebypiece.Thefirstthingtheproviderdoes,beyondincludinganyrequiredlibraries,istoinformChefthatitsupportswhy-run.ThisisamechanismthatChefprovidessothatresourcescanbemoreeasilytestedbyeffectivelynotwiringaresourcetoaprovider.Thisallowsdeveloperstotesttheirresources,inwhatiseffectivelyadry-runmode,beforerunningthemliveagainstasystem:

#Onlyrunasneeded

defwhyrun_supported?

true

end

Next,thereisanactionblock—thisregisterstheprovidedblockasthelogictobeexecutedforthespecifiedaction(inthiscase,:sync).Thishasthegeneralformsuchas:

action:<actionname>do

#Realworkinhere

end

Inthiscase,theonlysupportedactionissync,andsothereisonlyoneactionblock:

action:syncdo

Chef::Log.debug("Checking#{new_resource}forchanges")

fetch_from_s3(new_resource.source)do|raw_file|

Chef::Log.debug"copyingremotefilefromorigin#{raw_file.path}to

destination#{new_resource.destination}"

FileUtils.cpraw_file.path,new_resource.destination

end

new_resource.updated

end

Here,the:syncactionleveragesthefetch_from_s3method,whichyieldsalocalcopyofafileintheremotebucketonceithasbeendownloaded.Then,thefileiscopiedfromthetemporarylocationlocallyintothespecifieddestination.

ModifyingresourcesInsideofthisaction,youwillnoticethatthereisanactor,new_resource(whichisactuallyabuilt-inmethod).Thisdescribeswhatthestateofthenamedresourceshouldbewhentheproviderhascompleteditsexecutionforthespecifiedresource;thismayormaynotdifferfromthecurrentstateoftheresourceonthenode.Inthecaseofaninitialrun,new_resourcewillalmostcertainlybedifferentfromcurrent_resource,butthatmaynotalwaysbethecaseonsubsequentruns.

Asanexample,ifwehavearecipewiththefollowingS3bucketresourcedeclared:

s3_bucket"s3://mychefbucket/.resource"do

action:sync

skip["foo.txt","bar.txt]

destination"/opt/app_data"

access_key_idnode[:app][:aws_access_key]

secret_access_keynode[:app][:aws_secret_key]

ownernode[:app][:user]

groupnode[:app][:group]

mode"0755"

end

Then,thenew_resourceactorwouldhaveitsmembervariablespopulatedwiththeparameterspassedtothes3_bucketresource.Again,thisistheexpectedstateoftheresource,thewayitshouldbewhentheexecutionbytheprovideriscomplete.Inthiscase,whentheprovidercodeisexecuted,new_resource.destinationwillbe"/opt/app_data"andnew_resource.skipwillbealistof"foo.txt"and"bar.txt"andsoon.ThisallowsyoutopassdataintotheinstanceoftheresourceinthesamewaythatwaspossiblewiththePIPandTornadoapplicationdefinitions.

LoadinganexistingresourceOnethingthatislessobviousabouttheproviderscriptistheload_current_resourcemethodthatisnotcalledfromwithintheprovider.ThismethodisusedbyCheftofindaresourceonthenodebasedontheattributesthatareprovidedbytherecipe.Thisisusefultodetermineifanythingneedstobedonetobringanexistingresourceonthehostsuchasafile,auseraccount,oradirectoryoffiles,uptodatewiththedatathatisprovidedduringexecutionoftherecipe.

Itmightmakesensetoextendthisprovidertoprecomputethehashesofthefilesthatalreadyexistinthedirectoryon-diskasspecifiedbydestination.Thisway,theprovidercanbeupdatedtoonlydownloadanyremotefilesinS3thathaveadifferentfingerprintthanasimilarlynamedresourceondisk.Thispreventsunnecessaryworkfrombeingperformed,whichsavestime,bandwidth,andotherresources.

Here,however,itisalsousedtoensurethatanydependenciestodownloadfilesareinstalled;inthiscase,theAWSgemisrequiredtousetheS3client.Thisworksbecausetheload_current_resourcemethodgetscalledonearlytodeterminethecurrentstateoftheresource.Iftheresourcesarethesame,thentheproviderhasnothingtodo.ThecurrentimplementationjustclobberswhateverfilesarelocalwiththecontentsoftheS3bucket(moreofaone-waydownloadthanasync,really).

DeclaringthataresourcewasupdatedResourceshaveabuilt-inmethod,updated_by_last_action,whichiscalledinsidethe:syncactionallthetimeinthisexample.Thismethodnotifiestheresourcethatthenodewasupdatedsuccessfully.Thisshouldonlybesettotrueifeverythingwassuccessfullyupdated;failuresshouldnotmakethiscallunlesstheysetittofalse.Itisusefultoknowwhatresourceshavebeenupdatedforreportingorotherpurposes.Forexample,youcanusethisflagtoidentifywhatresourceshavebeenupdated:

moduleSimpleReport

classUpdatedResources<Chef::Handler

defreport

Chef::Log.info"Resourcesupdatedthisrun:"

run_status.updated_resources.eachdo|r|

Chef::Log.info"#{r.to_s}"

end

end

end

end

WorkingwithdatabagsThereareanumberofthingsyoucandowithdatabags.

SecuringyourdatabagsDatabagsarejustJSONdata,buttheyarestoredinthesystemasplaintext,withoutanysecurity.Theyarealsodownloadedontovarioushoststhroughoutthelifecycle,whichcanleadtoleakingofpotentiallysensitiveinformation.Fortunately,Chefhasamethodthatletsyousecurethisdatabyusingknife,alongwithsecretkeystokeepdataindatabagsencrypted.

SecretkeysEncryptingadatabagitemrequiresasecretkey;onewayofgeneratingasecretkeyistogeneratearandomnumberandusetheBase64encodingofthatnumberasthesecretkey.Thisshouldhaveanylineendingsremovedtoensureitworksproperlyonallplatforms,regardlessofplatform-specificlineendings.Hereisaquickwaytogenerateoneusingtheopensslcommandlinetoolcombinedwithtrtoremoveanylineendings:

$opensslrand-base64512|tr-d'\r\n'>~/.chef/data_bag_secret

EncryptingyourdataInordertoencryptyourdatabagitem,youmustuseknifeandpassthe--secretor--secret-fileflagstoknifewhencreatingtheitem.Forexample,tocreateadatabagcalledcredentialsandstoreanewentry,aws,insideit,youwouldusethefollowingcommand(makesureyousetyourEDITORenvironmentvariablefirst):

$knifedatabagcreatecredentialsaws--secret-file

~/.chef/data_bag_secret

Asmentionedbefore,youwillbepresentedwiththecontentsofyournewdatabagiteminyoureditor,unencrypted:

1{

2"id":"aws",·

3}

Here,wecanaddsomeproperties,suchasasecretkey:

1{

2"id":"aws",·

3"secret_key":"A21AbFdeccFB213f"

4}

Onceyousavethis,knifewilltellyouthatthenewdatabagwascreated,alongwiththenewdatabagiteminChef,justasitdidwiththeuserdataearlier.TheonlydifferencewillbethatthistimethedatastoredintheChefserverhasbeenencryptedusingthesymmetrickeyyouprovided:

$knifedatabagcreatecredentialsaws--secret-file

~/.chef/data_bag_secret

Createddata_bag[credentials]

Createddata_bag_item[aws]

Tocheckwhetheryournewlycreateddatabagentrywasencrypted,useknife,aswehave

before,toshowthecontentsofanitem:

$knifedatabagshowcredentialsaws

id:aws

secret_key:

cipher:aes-256-cbc

encrypted_data:

SG4z4jd4VAnJ4gG0wPcJWOX7H+ZNSxG5PH+n7EgHFV9e1SciVznjaAbzK61c

EW0/

iv:rKB0riCr84QhBkw+Wgc/5Q==

version:1

DecryptingyourdataInordertodecryptthedatainthedatabagitem,youneedtoprovidethesamesymmetrickeyasyouprovidedwhenyouencryptedit,usingthe--secretor--secret-fileargument,ascanbeseenhere:

$knifedatabagshowcredentialsaws--secret-file~/.chef/data_bag_secret

id:aws

secret_key:A21AbFdeccFB213f

Ifitwasn’talreadyobvioustoyou,makecertainyoudonotlosethisfile.Withoutyoursecretkeyorsecretfile,youwillnotbeabletodecryptthedatainyourdatabag.Itmaybeworthencryptingthesecretfilewithapassphraseifyouaregoingtobetransmittingittonontrustedlocationsaswell.

StoringkeysonnodesAnencryptionkeycanalsobestoredinanalternatefileonthenodesthatneedit,andyoucanspecifythepathlocationtothefileinsideanattribute;however,EncryptedDataBagItem.loadexpectstoseetheactualsecretasthethirdargumentratherthanapathtothesecretfile.Inthiscase,youcanuseEncryptedDataBagItem.load_secrettoslurpthesecretfilecontentsandthenpassthem:

#insideyourattributefile:

default[:app][:aws_creds_secret]="/opt/secret/aws.secret"

#Insideyourrecipe

aws_secret=Chef::EncryptedDataBagItem.load_secret

"#{node[:app][:aws_creds_secret]}"

aws_creds=Chef::EncryptedDataBagItem.load

"credentials","aws",aws_secret

aws_creds["secret_key"]

SearchingyourdataAswediscussedearlier,youcansearchthroughyourdatabagsusingBooleansearchlogic.Thispermitsyoutofindonlytheentriesinyourdatabagsthatyouneed.Thesamesearchquerylanguageisusedonthecommandlinewithknifeasitisinyourrecipes,sothatyoucantestyourqueriesonthecommandlinetoensurethattheyproducetherightresultsbeforeyouputtheminyourrecipes.Youcanalsosearchthroughotherresourcesaswell,notjustdatabags.

SearchingyourdatabagswithknifeTheknifetoolusesthesearchcommandtosearchthroughyourdatabags.Thegeneralsyntaxis:

knifesearch<source>"<searchcriteria>"

SearchingyourdatabagsfromarecipeInsidearecipe,thesearchmethodisusedtosearchthroughadatabag.Thesyntaxforthisis:

search(:source,"searchcriteria")

QueryingyourdataThesearchqueryformatisreasonablystraightforwardandlookslikemostothersearchenginesthatsupporttheBooleanlogic.

Searchesonattributescomeintheformofkey:value;soforexample,ifyouwantedtofindalloftheuserswhoweredwarvesfromourearlierdatasets,youcanusethesearchquery:

knifesearchusers"groups:dwarves"

NegatingasearchtermcanbeaccomplishedbyplacingNOTinfrontofthesearchterm.Forexample,alluserswhoarenothobbitswillbe:

knifesearchusers"NOTgroups:hobbits"

YoucanalsouseanORmodifier:

knifesearchusers"groups:elvesORgroups:hobbits"

Thislastsearchcriteriawouldyieldtheuserslegolas,samwise,andfrodoasFrodoandSamwiseareinthegroupcalledhobbitsandLegolasisintheelvesgroup.

Whilecombiningsearchterms,youcanlogicallyANDthemtogetheraswell.Forexample,alluserswithaGIDstartingwith20,whocontainthegroupelvescanbefoundusingthefollowingquery:

knifesearchusers"groups:elvesANDgid:20*"

Youcansearchyournodeswiththesamequerylanguage—inordertofindallnodesthatarerunningsomeformofwindows,youcansearchfortheplatformbeinganythingthat

startswithwin:

knifesearchnode"platform:win*"

ThiswillyieldallWindowshosts(resultshavebeenshortenedabit):

4itemsfound

NodeName:i-13d0bd4f

Roles:

Platform:windows6.2.9200

NodeName:WIN-CJDQ9DEOJFK

Roles:umbraco_cms

Platform:windows6.2.9200

NodeName:00c0ff3300

Roles:

Platform:windows6.2.9200

NodeName:rs-5889646228538071

Roles:

Platform:windows6.2.9200

Or,youcansearchforallnodesthatarerunningwindowsandcontaintheroleumbraco_cms:

knifesearchnode"platform:win*ANDrole:umbraco_cms"

Or,ifyouwantedtoeliminatethosenodesthatruntheUmbracoCMS,youcaneasilyinverttherolecondition:

knifesearchnode"platform:win*ANDNOTrole:umbraco_cms"

BecauseChefusesApacheSolrtosearchitsdata,youcanrefertotheApacheSolrdocumentationonbuildingmoreadvancedquerylogicathttp://wiki.apache.org/solr/SolrQuerySyntax.

ManagingmultiplemachineswithsearchqueriesThesearchcriteriacanbeusedforallsortsofplaces:inrecipes,onthecommandline,throughAPIcalls,andmore.OneveryinterestingapplicationisbeingabletousethesearchquerytoSSHtomultiplemachinestoperformcommandsinparallel:

knifessh"fqdn:*.east.mycorp.comANDplatform:ubuntu""chef-client"-x

app_user

ThiswillcontacttheChefserverandaskforthenodesthatmatchthegivenquerystring(machineswhoseFQDNmatchesthewildcardexpression"*.east.mycorp.com"andthatarerunningUbuntu)andthenconnecttothemviaSSHastheuserapp_userandrunthechef-clientcommandoneachofthem.Again,youcanrestrict(orexpand)theserverlistbyusingamore(orless)specificquery.

Onceyouhavemasteredthisaspectofusingknife,youcanlearnmoreaboutitssupportforexecutingmultipleconnectionsconcurrentlyandeveninteractwithterminalmultiplexerssuchasscreenandtmux.

SummaryChefhaslotsofmechanismstobuildadvancedautomation,includingbuildingyourowndefinitions,resources,andproviders,aswellasstoringandaccessingcomplexconfigurationdataandevensecurelyencryptingit.Thischapterhasshownyouhowtomanagedataindatabags(includingencrypteddata),useChef’sadvancedsearchenginetofindandmanipulatedatainyoursystemfromthecommandlineandinrecipes,aswellasdevelopdefinitionsforreusablerecipedevelopment,andevenbuildcustomresourcesandprovidersforuseinyourcookbooks.

Inthenextchapter,wewillcoversomemoreadvancedwaystouseChef,includinginteractingwiththeChefshell,automationandintegrationwithChefusingscriptsandAPIs,externaltoolsandresources,advancedtestingincludingintegrationtesting,andusingChef-soloandVagranttomanageyourdevelopmentenvironments.

Chapter8.ExtrasYouNeedtoKnowThischapterwillcoverhowtouseCheftobuildcustombootstrapscriptsforsystems,enhancedcommand-linetoolconcepts,leverageChefforautomation,integration,andsecurelystoresensitivedatainthesystem.Sometopicsthatwillbecoveredinthischapterinclude:

UsingChef-solowithVagrantInteractingwiththeChefshellDebuggingrecipesAdvancedcommand-lineusageAutomatingandintegratingwithChefMoretestingmethodologies

VagrantandChef-soloVagrantisaveryusefultooltobuilddevelopmentenvironments,whereitprovidestoolstobuildvirtualmachinesthatcontaineverythingyouneedtogetstartedwithbuildingsoftware.Consider,foramoment,workingonateamthatbuildssoftwareandreliesonaservice-orientedarchitecture(SOA),andthissoftwareiscomposedofanumberofdifferentservices.Inorderforittowork,youmayberequiredtoinstallandconfigureallofthedependentservicestoevenbeginworkingonapartofthesystem;thiscouldbeatime-consuminganderror-proneexerciseforevenseasoneddevelopers.Nowimaginethatallyouhadtodowasdownloadaconfigurationfileandexecutevagranttodoitforyou—thisistheworldofVagrant.

OneoftheinterestingfacetsofVagrantisthatithassupporttoprovisionnewinstancesusinganumberofdifferentmechanisms.Currently,thislistincludes10orsodifferenttools,butthemostinterestingtwoareChef-soloandChefclient.Bynow,youshouldbecomfortablewithhowyoumightprovisionavirtualmachineusingtheChefclient;it’snotmuchdifferentthanprovisioninganEC2instanceoradedicatedserver.However,wehaven’tdiscussedusingChef-solomuchyet,sothisisagoodtimetolearnmoreaboutit.

InstallingVagrantHistorically,VagrantwasinstalledviaRubyGems;thisisnolongerthecase,andifyouhaveanolderversioninstalledasagem,itisrecommendedthatyouremoveitbeforeinstallingVagrant.Installersforallsupportedplatforms(OSX,Windows,andLinux)areavailableatthefollowingURL:

http://www.vagrantup.com/downloads

IfyouarenewtoVagrant,theninadditiontoinstallingVagrant,youwillwanttoinstallVirtualBoxforsimplicity,asVagranthasbuilt-insupportforVirtualBox.VagrantdoessupportotherproviderssuchasVMWareandAWS,butitrequirespluginsthatarenotdistributedwiththecoreVagrantinstallationinorderforthemtowork.

OnceyouhaveinstalledVagrantandVirtualBox,thenyoucancontinueonwiththefollowingexamples.

ProvisioninganewhostwithVagrantProvisioninganewvirtualinstancerequiresthatyoubuildaVagrantconfigurationfilecalledVagrantfile.Thisfileservestwopurposes:todenotethatthedirectoryisaVagrantproject(similartohowaMakefileindicatesaprojectthatisbuiltwithMake),andtodescribethevirtualmachinethatisbeingrun,includinghowtoprovisionit,whatoperatingsystemtouse,wheretofindthevirtualimage,andsoon.Becausethisisjustaplaintextfile,youcanincludeitalongwithanyauxiliaryfilesrequiredtobuildtheimagesuchascookbooks,recipes,JSONfiles,installers,andsoon,andcommitittothesourcecontrolforotherstouse.

Inordertobegin,youwillwanttocreateadirectorythatwillhouseyournewVagrantproject.OnUnix-likesystems,wewouldbootstrapourprojectsimilarlytothefollowingcommand:

mkdir-p~/vagrant/chef_solo

cd~/vagrant/chef_solo

Windowshostswillbethesameexceptfordifferentpathsandchangesinmethodsofdirectorycreation.Oncethisstepiscomplete,youwillneedtocreateaskeletonconfigurationlocatedin~/vagrant/chef_solo/Vagrantfile.Thisfilecanbegeneratedusingvagrantinit,butwewouldnotwanttousethecontentsofthegeneratedfile;so,wewillskipthatstepandmanuallyconstructourVagrantfileinstead(withasimpleone-lineconfigurationthatusesabaseimageofUbuntu13.10).YourVagrantfileshouldlooklikethefollowingcode:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

end

Here,"2"istheAPIversion,whichiscurrentlyVersion2.0asofthiswriting,andtheconfiguredbaseimage(orbox)istheUbuntuTrusty(14.04)64-bitimage.Beforeyouusethisbaseimage,itneedstobedownloadedtoyourlocalmachine;youcanaddittoVagrantusingtheboxaddcommand:

vagrantboxaddubuntu/trusty64

TipThisstepwilltakeafewminutesonafastconnection,sobepreparedtowaitwhilethiscompletesifyouarefollowingalonginteractively.Alsonotethatifyouskipthisstep,thebaseimagewillautomaticallybedownloadedwhilerunningvagrantuptostartyourvirtualmachine.

Forfuturereferences,ifyouwanttofindalternativeOSimagestouseforyourVagrantmachines,youshouldlookatVagrantCloud(https://vagrantcloud.com/),whereyoucanfindanumberofotherfreelyavailablebaseimagestodownloadforusewithVagrant.

BootingyourVagrantimageOnceyourbaseimagehascompleteddownloading,youwillusethevagrantupcommandtobootupanewvirtualmachine.Bydoingthis,youwillinstructVagranttoreadtheVagrantfileandbootanewinstanceofthebaseimage:

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'ubuntu/trusty64'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'ubuntu/trusty64'isuptodate…

==>default:SettingthenameoftheVM:

chef_solo_default_1402875519251_51266

==>default:Clearinganypreviouslysetforwardedports…

==>default:Clearinganypreviouslysetnetworkinterfaces…

==>default:Preparingnetworkinterfacesbasedonconfiguration…

default:Adapter1:nat

==>default:Forwardingports…

default:22=>2222(adapter1)

==>default:BootingVM…

==>default:Waitingformachinetoboot.Thismaytakeafewminutes…

default:SSHaddress:127.0.0.1:2222

default:SSHusername:vagrant

default:SSHauthmethod:privatekey

default:Warning:Connectiontimeout.Retrying…

==>default:Machinebootedandready!

==>default:CheckingforguestadditionsinVM…

==>default:Mountingsharedfolders…

default:/vagrant=>/Users/jewart/Temp/vagrant/chef_solo

Asyoucanseefromtheoutput,Vagrantperformedthefollowingthings:

Usedthebaseimageubuntu/trusty64ConfiguredVirtualBoxtouseaNATadapter,mappingport22to2222StartedtheVMinheadlessmode(suchthatyoudon’tseetheVirtualBoxGUI)Createdauser,vagrant,withaprivatekeyforauthenticationMountedasharedfoldermapping/vagrantontheguesttotheVagrantworkspaceonthehost

Nowthatyouhavearunningguest,youcancontrolitbyrunningvagrantcommandsfrominsideofthevagrantworkspace(~/vagrant/chef_solo);forexample,youcanSSHintoitusingthefollowingcommand:

vagrantssh

Andyoucandestroytherunninginstancewiththefollowingcommand:

vagrantdestroy

GoaheadandSSHintoyournewguestandpokearoundalittlebit—youwillnoticethatitlooksjustlikeanyotherUbuntu14.04host.Onceyouaredone,usedestroytodestroyitsothatyoucanlookathowtoprovisionyourVagrantimageusingChef-solo.It’simportanttoknowthatifyouusedestroyonyourguest,changestoyourVagrantimagearenotpersisted;so,anychangesyouhavemadeinsideitwillnotbesavedandwillnot

existthenexttimeyouusevagrantuptostarttheVM.

CombiningVagrantwithChef-soloInourpreviousexample,ourVagrantfilesimplydeclaredthatourguestreliedontheubuntu/trusty64imageasthebaseimageviatheconfig.vm.boxproperty.Next,wewilllookathowtoextendourconfigurationfiletousetheChef-soloprovisionertoinstallsomesoftwareonourguesthost.Here,wewilluseChef-solotoinstallPostgreSQL,Python,andawebapplicationinsideoftheguest.

YouwillprobablynoticethattheconfigurationsectionsintheVagrantfilelooksortoflikeresourcesinChef—thisisbecausetheybothleverageRubyblockstoconfiguretheirresources.SowithVagrant,inordertospecifytheprovisioningmechanismbeingused,theconfig.vm.provisionoptionissettothedesiredtool.Here,wewilluseChef-solo,whichisnamed"chef_solo";so,wewillextendourVagrantfiletoindicatethis:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

config.vm.provision"chef_solo"do|chef|

#...Chefspecificsettingsblock

end

end

UnderstandingthelimitationsofChef-soloForthemostpart,Chef-solooperatesalotlikethetraditionalclient-servermodeofchef-client.TheprimarydifferencesresultfromthefactthatChef-solodoesnotinteractwithacentralChefserverandthereforelackssupportforthefollowing:

NodedatastorageSearchindexesCentralizeddistributionofcookbooksAcentralizedAPIthatinteractswithandintegratesinfrastructurecomponentsAuthenticationorauthorizationPersistentattributes

Asaresult,ifyouarewritingrecipestobeusedwithChef-solo,youwillbeunabletorelyonsearchfornodes,roles,orotherdataandmayneedtomodifythewayyoufinddataforyourrecipes.Youcanstillloaddatafromdatabagsforcomplexdata,buttheywillnotbecentrallylocated;rather,theywillbelocatedinanumberofJSONfilesthatcontaintherequireddata.

ConfiguringChef-soloThereareanumberofoptionsavailablefortheChef-soloprovisionerinVagrant.Forthemostup-to-datedocumentationofVagrant,besuretovisittheofficialVagrantdocumentationsiteathttp://docs.vagrantup.com/v2/.

MostoftheconfigurationoptionsarewaystoprovidepathstovariousChefresourcessuchascookbooks,databags,environments,recipes,androles.AnypathsspecifiedarerelativetotheVagrantworkspaceroot(wheretheVagrantfileislocated);thisisbecausethesearemountedintheguestunder/vagrantandaretheonlywaytogetdataintothe

hostduringthebootstrapphase.Theoneswewillbeusingare:

cookbooks_path:Thisconsistsofasinglestringoranarrayofpathstothelocationwherecookbooksarestored.Thedefaultlocationiscookbooks.data_bags_path:Thisconsistsofapathtodatabags’JSONfiles.Thedefaultpathisempty.roles_path:ThisconsistsofanarrayorasinglestringofpathswhererolesaredefinedinJSON.Thedefaultvalueisempty.

Inourcase,wewillbereusingourexamplecookbooksfromtheearlierchapter.YoucanfetchthemfromGitHubathttp://github.com/johnewart/chef_cookbook_files;eitherdownloadtheZIPfileorclonethemusingGitlocally.Onceyouhavedonethat,copycookbooks,roles,anddata_bagsfromthearchivetoyourVagrantworkspace.ThesewillbetheresourcesthatyouwilluseforyourVagrantimageaswell.InordertotellVagrant’sChef-soloproviderhowtofindthese,wewillupdateourVagrantfileagaintoincludethefollowingconfiguration:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

config.vm.provision"chef_solo"do|chef|

chef.cookbooks_path="cookbooks"

chef.roles_path="roles"

chef.data_bags_path="data_bags"

end

end

TellingChef-solowhattorunInsideoftheprovisionblock,wehaveaChefobjectthateffectivelyrepresentsaChefclientrun.Thisobjecthasanumberofmethods(suchasthepathsettingswealreadysaw),oneofwhichistheadd_recipemethod.Thisallowsustomanuallybuildourrunlistwithoutrequiringrolesordatabagsandcanbeused,asshowninthefollowingexample,toinstallthePostgreSQLserverwithnospecialconfiguration:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

config.vm.provision"chef_solo"do|chef|

chef.cookbooks_path="cookbooks"

chef.roles_path="roles"

chef.data_bags_path="data_bags"

#Buildrunlist

chef.add_recipe"postgresql::server"

end

end

ThiswilltellVagrantthatwewanttouseourdefineddirectoriestoloadourresources,andwewanttoaddthepostgresql::serverrecipetoourrunlist.Becausecookbooksarebydefaultexpectedtobein[vagrantroot]/cookbooks,wecanshortenthisexampleasshowninthefollowingcode,aswearenotyetusingrolesordatabags:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

config.vm.provision"chef_solo"do|chef|

chef.add_recipe"postgresql::server"

end

end

UsingrolesanddatabagswithChef-soloAsyouarealreadyawarebynow,wemaywanttoperformmorecomplexconfigurationofourhosts.Let’stakealookathowtousebothrolesanddatabagsaswellasourcookbookstodeployourPythonwebapplicationintoourVagrantguestsimilartohowwedeployedittoEC2:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

config.vm.provision"chef_solo"do|chef|

chef.cookbooks_path="cookbooks"

chef.roles_path="roles"

chef.data_bags_path="data_bags"

#Buildrunlist

chef.add_role("base_server")

chef.add_role("postgresql_server")

chef.add_role("web_server")

end

end

Justlikethecookbookspath,therolespathisrelativetotheprojectrootifarelativepathisgiven.

InjectingcustomJSONdata

AdditionalconfigurationdataforChefattributescanbepassedintoChef-solo.ThisisdonebysettingthejsonpropertywithaRubyhash(dictionary-likeobject),whichisconvertedtoJSONandpassedintoChef:

Vagrant.configure("2")do|config|

config.vm.provision"chef_solo"do|chef|

#...

chef.json={

"apache"=>{

"listen_address"=>"0.0.0.0"

}

}

end

end

Hashes,arrays,andsooncanbeusedwiththeJSONconfigurationobject.Basically,anythingthatcanbeturnedcleanlyintoJSONworks.

Providingacustomnodename

Youcanspecifyacustomnodenamebysettingthenode_nameproperty.Thisisusefulforcookbooksthatmaydependonthisbeingsettosomesortofvalue.Forexample:

Vagrant.configure("2")do|config|

config.vm.provision"chef_solo"do|chef|

chef.node_name="db00"

end

end

GettingtoknowtheChefshellTheChefshell,previouslycalledshef,providesaninteractivetoolorread-eval-print-loop(REPL)toworkwithChefresources.MuchinthesamewayIRBoranyotherlanguage’sREPLshellworks,chef-shellisawaytointeractwithknife.Thisishandyforexperimentingwithresourceswhilewritingrecipessothatyoucanseewhathappensinteractivelyratherthanhavingtouploadyourcookbooktoaserverandthenexecutingthechef-clientonatargetnode.Additionally,theChefshellprovidesaresourcetoaddbreakpointstorecipeexecutionsothatitcanbeusedtodebugrecipeexecution,whichisaveryhandyfeature.

UsingtheChefshellAsof11.x,shefhasbeenreplacedwithchef-shellandcanbeusedinthreedifferentmodes:standalone,solo,andclientmode.Eachofthesehasaslightlydifferentsetoffunctionalitiesandexpectedusecases.

ThestandalonemodeThestandalonemodeisusedtorunChefinaninteractivemodewithnothingloaded;thisisalmostlikerunninganREPLsuchasirborpythononthecommandline.Thisisalsothedefaultbehaviorofchef-shellifnothingisspecified.

ThesolomodeThesolomodeisinvokedusingthe-sor--solocommand-lineflagandisawaytousechef-shellasachef-soloclient.Itwillloadanycookbooksusingthesamemechanismthatchef-solouserswould,anditwilluseanychef-soloJSONfileprovidedtoitusingthe-jcommand-lineoption.

Thefollowingareexamplesofusingthesolomode:

chef-shell-s

chef-shell-s-j/home/myuser/chef/chef-solo.json

TheclientmodeTheclientmodeisenabledwiththe-zor--clientcommand-lineflag;thismodecauseschef-shelltoactasthoughyouinvokedchef-clientonthehost.Theshellwillreadthelocalclientconfigurationandperformthenormaldutiesofchef-client:connectingtoyourChefserveranddownloadinganyrequiredrunlists,attributes,andcookbooks.However,itwillallowforinteractiveexecutionsothatitispossibletodebugordiagnoseissueswithrecipesontheendhost.Whenusingtheclientmode,youcanuseanalternateconfigurationfilewiththe-ccommand-lineoption,orspecifyadifferentChefserverURLviathe-scommand-lineoption.

Theexampleusesthefollowing:

chef-shell--client-c/etc/chef/alternate.conf

chef-shell--client-shttp://test.server.url:8080/

InteractingwiththeChefserverusingtheshellTheChefshellprovidesyouwiththeabilitytointeractwiththeserverquicklyinthesamewayyouwoulduseknife,butwithouttheoverheadoftypingknifesearchnode…orknifenodelist,andsoon.ItisaveryconvenientwaytoquerythedatastoredintheChefserverinteractively.Inordertointeractwiththeserverfromyourworkstation,youneedtomakesurethatyourshell’sconfigurationfile,locatedin~/.chef/chef_shell.rb,isconfiguredproperly.Ifyouareconnectingwithchef-shellfromanode,thentheconfigurationin/etc/chef/client.rb(orsimilaronWindows)willbeusedinstead.

Thisfile,similartotheknife.rborclient.rbfile,containstherequiredcertificatedataandconfigurationdatatoconnecttotheChefserver.Anexampleconfigurationfilewillresemblethefollowing,withpaths,organization,andclientnamesupdatedaccordingly:

node_name'myorg'

client_keyFile.expand_path('~/.chef/client.pem')

validation_keyFile.expand_path('~/.chef/validator.pem')

validation_client_name"myorg-validator"

chef_server_url'https://api.opscode.com/organizations/myorg'

Allofthesefilesarepresentifyourknifeinstallationisoperational,andtheconfigurationfilecloselyresemblesthatofknife.rb—ifyouneedvaluesfortheseonyourworkstation,takealookatthe~/.chef/knife.rbfile.Onceyouhaveconfiguredyourshell,youcanpassthe-zcommand-lineflagtoconnectasthechef-clientwould:

[jewart]%chef-shell-z

loadingconfiguration:/Users/jewart/.chef/chef_shell.rb

Sessiontype:client

Loading…...resolvingcookbooksforrunlist:[]

SynchronizingCookbooks:

done.

Thisisthechef-shell.

ChefVersion:11.12.8

http://www.opscode.com/chef

http://docs.opscode.com/

run'help'forhelp,'exit'or^Dtoquit.

Ohai2ujewart@!

chef>

InteractingwithdataFromhere,youcaninteractwiththeChefserverinavarietyofways,includingsearching,modifying,anddisplayinganydataelements(roles,nodes,databags,environments,cookbooks,andclients),performingaclientrun(includingsteppingthroughit,onestepatatime),assumingtheidentityofanothernode,andprintingtheattributesofthelocalnode.Forexample,listingtherolesontheChefservercanbeperformedwiththeroles.allmethod,shownasfollows:

chef>roles.all

=>[role[umbraco_cms],role[umbraco],role[base_server],role[web_server],

role[postgresql_server]]

SearchingyourdataSearchingthedataelementsisalsosupported,aseachdatatypehasafindmethodattachedtoit.Thefindmethodtakesamapoftheattributeandpatterntolookforandreturnstheresults.Forexample,youcanfindallrolesontheChefserverthatbeginwith"um"withthefollowingcommand:

chef>umbraco_roles=roles.find(:name=>"um*")

=>[role[umbraco_cms],role[umbraco]]

EditingyourdataAnyobjectinyourChefservercanbeediteddirectlyfromtheChefshellusingtheeditcommandfrominsidetheshell.ThiswillinvokeyourfavoriteeditortoedittherawJSONoftheobjectinquestion,whichprovidesamoredirectmechanismoverusingknife(node|role|databag)editonthecommandline,asyoucanquicklymanipulateanumberofrecordsalotmoreeasily.Forexample,toeditalloftherolesthatcontainthename"apache"andsavetheresults,youcanusethefollowingRubycode:

chef>apache_roles=roles.find(:name=>"*apache*")

>[...somelist…]

chef>apache_roles.eachdo|r|

chef>updated=editr

chef>updated.save

chef>end

Thiswillfindallroleswhosenamecontains"apache".Thenforeachrecord,edittheJSON,storingtheresultsinvariablenamedupdated,andthensavethatrecordbacktotheChefserver.

TransformingdataInthisway,youcaninteractwithanyoftheresourcesthatareavailabletoyou,allowingyoutoquicklyfindandmanipulateanydatastoredinChefdirectlyusingtheRubycode.Forexample,tofindallclientswithagivenstringintheirnameanddisabletheiradministrativeaccess,youcanusethefollowingcode:

clients.transform("*:*")do|client|

ifclient.name=~/bad_user/i

client.admin(false)

true

else

nil

end

end

TipUsecautionwhentransformingyourdatafromtheChefshell;itisanincrediblypowerfultool,butitseffectsaredestructive.Thesechangesarenotreversible(atleastnotwithoutforethoughtorbackups)andcoulddamageyourdataifyouarenotcareful.Forexample,

ifthepreviouscodewastranscribedincorrectly,itcouldpotentiallyrenderallusersunabletoadministerthesystem.

ExecutingrecipeswithChefshellTwogreatfeaturesofchef-shellaretheabilitytorewindarun(tostepbackwards)andtobeabletostepforwardintherunoneresourceatatime.Asanexample,let’slookathowtodefineasimplerecipeinchef-shellinteractivelyandthenrunit,startitover,andstepthroughit.

First,let’sfireupchef-shellwiththefollowingcommand:

[jewart]%chef-shell

loadingconfiguration:none(standalonesession)

Sessiontype:standalone

Loading…...done.

Thisisthechef-shell.

ChefVersion:11.12.8

http://www.opscode.com/chef

http://docs.opscode.com/

run'help'forhelp,'exit'or^Dtoquit.

Ohai2ujewart@!

chef>

Thechef-shellpromptwillchangebasedonthestateyouarein.Ifyouareworkingwitharecipe,thepromptwillchangetobechef:recipe>.

CreatingarecipeintheshellTheChefshellhasanumberofmodes—recipemodeandattributemode.Recipemodeisactivatedwhenworkingwithrecipesandwillbewhatweusehere.Inordertoactivateit,typerecipe_modeattheprompt:

chef>recipe_mode

chef:recipe>

Here,wewillcreateresourcestocreateafileinthecurrentdirectoryinteractivelyusingafileresourcewithnoassociatedconfigurationblock,onlythename:

chef:recipe>file"foo.txt"

=><file[foo.txt]@name:"foo.txt"@noop:nil@before:nil@params:{}

@provider:Chef::Provider::File@allowed_actions:[:nothing,:create,

:delete,:touch,:create_if_missing]@action:"create"@updated:false

@updated_by_last_action:false@supports:{}@ignore_failure:false

@retries:0@retry_delay:2@source_line:"(irb#1):1:in'irb_binding'"

@guard_interpreter::default@elapsed_time:0@resource_name::file@path:

"foo.txt"@backup:5@atomic_update:true@force_unlink:false

@manage_symlink_source:nil@diff:nil@sensitive:false@cookbook_name:

nil@recipe_name:nil>

TipOnethingtonotehereisthattheshellwillprintouttheresultsofthelastoperationexecutedintheshell.ThisispartofanREPLshell’simplicitbehavior;itistheprintpart

ofREPL:inputisreadandevaluated,thentheresultsareprintedout,andtheshellloopstowaitformoreinputfromtheuser.Thiscanbecontrolledbyenablingordisablingtheechostate;echooffwillpreventtheprintedoutputandechoonwillturnitbackon.

Itiscriticaltonotethat,atthispoint,nothinghasbeenexecuted;wehaveonlydescribedafileresourcethatwillbeacteduponiftherecipeisrun.Youcanverifythisbymakingsurethatthereisnofilenamedfoo.txtinthedirectoryyouexecutedchef-shellfrom.Therecipecanberunbyissuingtherun_chefcommand,whichwillexecuteallofthestepsintherecipefromstarttofinish.Hereisanexampleofthis:

chef:recipe>run_chef

INFO:Processingfile[foo.txt]actioncreate((irb#1)line1)

DEBUG:touchingfoo.txttocreateit

INFO:file[foo.txt]createdfilefoo.txt

DEBUG:foundcurrent_mode==nil,sowearecreatinganewfile,updating

mode

DEBUG:foundcurrent_mode==nil,sowearecreatinganewfile,updating

mode

DEBUG:foundcurrent_uid==nil,sowearecreatinganewfile,updating

owner

DEBUG:foundcurrent_gid==nil,sowearecreatinganewfile,updating

group

DEBUG:foundcurrent_uid==nil,sowearecreatinganewfile,updating

owner

INFO:file[foo.txt]ownerchangedto501

DEBUG:foundcurrent_gid==nil,sowearecreatinganewfile,updating

group

INFO:file[foo.txt]groupchangedto20

DEBUG:foundcurrent_mode==nil,sowearecreatinganewfile,updating

mode

INFO:file[foo.txt]modechangedto644

DEBUG:selinuxutilitiescannotbefound.Skippingselinuxpermission

fixup.

DefiningnodeattributesJustasinanyrecipe,attributescanbeusedintherecipesdefinedintheshell.However,inthestandalonemode,therewillbenoattributesdefinedinitially;soloandclientmodeswilllikelyhaveattributesdefinedbytheirJSONfileortheChefserver,respectively.Inordertointeractwiththecurrentlydefinedattributes,wemustswitchbetweentherecipemodeandattributemode.Thisisachievedusingtheattributes_modecommandasshowninthefollowingcode:

chef:recipe>attributes_mode

chef:attributes>

Herewecanperformtwoprimaryoperations:gettingandsettingnodeattributes.ThesearewaysofmodifyingthevaluesthatareaccessedfrominsidethenodeMashinarecipe.

TipRememberthatthenode’sattributesareaccessedasaMash,akey-insensitivehashthatallowsyoutointerchangestringkeyswithsymbolkeys.TheMashclassisnotabuilt-in

structureinRuby—itisprovidedbyChefforconveniencesothathashkeyscanbeeithersymbolsorstringsandhavethesameeffect.

Settingattributes

Settingattributesisachievedusingthesetcommand,whichhasthefollowingform:

set[:key]=value

Here,:keycanbeasingle-levelkeyoramultilevelkeysimilartoanyentryintheattributes/default.rbfile.Asanexample,wecanconstructanapplicationconfigurationusingthefollowing:

set[:webapp][:path]="/opt/webapp"

set[:webapp][:db][:username]="dbuser"

set[:webapp][:db][:password]="topsecret"

set[:webapp][:user]="webuser"

set[:postgresql][:config][:listen]="0.0.0.0"

Anyparentkeysthatarenon-existentareimplicitlycreatedonthefly,soyoudonotneedtodosomethinglikethefollowing:

set[:webapp]={}

set[:webapp][:path]="/opt/webapp"

Accessingattributes

Inordertodisplayanattributewhenintheattributesmode,simplytypeinthenameofthekeyyouareinterestedin.Forexample,ifyouhadexecutedthesetcommandslistedpreviously,thenaskingforthewebapphashisassimpleastypingwebapp,asfollows:

chef:attributes>webapp

=>{"path"=>"/opt/webapp","db"=>{"username"=>"dbuser",

"password"=>"topsecret"},"user"=>"webuser"}

However,ifyouwishtoaccessthesewhenintherecipemode,theyareaccessedthroughthenodehash,asshownhere:

chef:attributes>recipe_mode

=>:attributes

chef:recipe>node[:webapp]

=>{"path"=>"/opt/webapp","db"=>{"username"=>"dbuser",

"password"=>"topsecret"},"user"=>"webuser"}

Theycanbeusedviathenodehashinjustthesamewayyouwouldusetheminarecipe.Ifyouwanttoconstructafileblockthatcreatedafoo.txtfilelocatedintheinstallpathofourwebapphash,youcaneasilyusethefollowingexampleinsideyourshell:

file"#{node[:webapp][:path]}/foo.txt"

Thismakeswritingrecipesusingtheinteractiveshellfeelexactlythesameaswritingrecipefiles.

UsingconfigurationblocksAresourceinarecipefilecanhaveaRubyblockwithattributes,andyoucandothisin

chef-shellinexactlythesamefashion.Simplyinsertdoaftertheresourcenameandtheshellwillbehaveasamultilineeditor,allowingyoutocompletetheblock.Thefollowingexampledemonstratesprovidingacontentattributetoafileresourceinthismanner:

chef:recipe>file"not_empty.txt"do

chef:recipe>content"Notempty!"

chef:recipe?>end

=><file[not_empty.txt]@name:"not_empty.txt"@noop:nil@before:nil

@params:{}@provider:Chef::Provider::File@allowed_actions:[:nothing,

:create,:delete,:touch,:create_if_missing]@action:"create"@updated:

false@updated_by_last_action:false@supports:{}@ignore_failure:false

@retries:0@retry_delay:2@source_line:"(irb#1):2:in'irb_binding'"

@guard_interpreter::default@elapsed_time:0@resource_name::file@path:

"not_empty.txt"@backup:5@atomic_update:true@force_unlink:false

@manage_symlink_source:nil@diff:nil@sensitive:false@cookbook_name:

nil@recipe_name:nil@content:"Notempty!">

Notethatwhentheshellprintedoutthepreviousfileresource,@contentwasnotpresent.Here,everythingbutthenameremainsthesame,andthereisanadditionalpropertyinsidetheobject,@content,asspecifiedinourattributesblock.

InteractivelyexecutingrecipesRunningarecipestepbystepisagoodwayofslowingdowntheexecutionofarecipesothatthestateofthesystemcanbeinspectedbeforeproceedingwiththenextresource.Thiscanbeincrediblyusefulbothfordebugging(aswillbediscussedlater)andfordevelopingandexploringresources.Itgivesyouachancetoseewhathashappenedandwhatsideeffectsyourrecipehasastherecipeisexecuted.Toachievethis,theChefshellallowsyoutorewindyourrecipetothestartandrunfromthebeginning,executeyourrecipeonestepatatime,andresumeexecutionfromthecurrentpointtotheend.

RestartingourChefshell,let’stakealookathowwecanusethis:

recipe_mode

echooff

file"foo.txt"

file"foo.txt"do

action:delete

end

file"foo.txt"do

content"Foocontent"

end

Hereourrecipeisquitesimple—createanemptyfile,foo.txt,removeit,andthenrecreateitwith“Foocontent”.Ifweexecuteourrecipeusingrun_chef,theshellwillperformalltheoperationsinonepasswithoutstoppingandwillnotallowustocheckwhetherthedeleteactionoccurred.Instead,wecanrunourrecipeandthenrewindandusethechef_run.stepmethodtointeractivelywalkthroughourrecipe:

chef:recipe>run_chef

...executionoutput…

chef:recipe>echoon

=>true

chef:recipe>chef_run.rewind

=>0

chef:recipe>chef_run.step

INFO:Processingfile[foo.txt]actioncreate((irb#1)line3)

=>1

chef:recipe>chef_run.step

INFO:Processingfile[foo.txt]actiondelete((irb#1)line4)

INFO:file[foo.txt]backedupto/var/chef/backup/foo.txt.chef-

20140615175124.279917

file[foo.txt]deletedfileatfoo.txt

=>2

chef:recipe>chef_run.step

INFO:Processingfile[foo.txt]actioncreate((irb#1)line7)

INFO:file[foo.txt]createdfilefoo.txt

INFO:file[foo.txt]updatedfilecontentsfoo.txt

INFO:file[foo.txt]ownerchangedto501

INFO:file[foo.txt]groupchangedto20

INFO:file[foo.txt]modechangedto644

=>3

Asyoucansee,herewewereabletorewindourrecipebacktothefirstinstruction(position0,astheresultofchef_run.rewindindicates),andthenwalkthrougheachresourcestepbystepusingchef_run.stepandseewhathappened.Duringthisrun,youcaneasilyopenaterminalafteryourewindtherecipe,deletethefoo.txtfilefromthepreviousrun,andcheckthatinitiallythereisnofoo.txtfile,thenstepthroughthenextcommandintherecipe,validatethatthereisanemptyfoo.txtfile,andsoon.Thisisaverygoodwaytolearnhowresourcesworkandtoseewhattheydowithouthavingtoformalizeyourrecipeinacookbook,provisionandbootstrapahost,andsoon.

DebuggingwiththeChefshellDebuggingisachievedintwodifferentwaysusingchef-shell:steppinginteractivelythrougharecipeorusingaspecialbreakpointresourcethatisonlyusedbychef-shell.Runningrecipesinteractivelystepbystepisgoodtobuildrecipeslocally;experimentwithresourcestodeterminetheeffectofcertainattributes,actionsandnotifications;ortoinspectthestateofthesystemaftereachresourcehasbeenactedupon.Thebreakpointsallowyoutoinjectveryspecificstoppingpointsintotheclientrunsothattheworldcanbeinspectedbeforecontinuing.Typically,onceabreakpointisencountered,youwillwanttostepthroughtheexecutionofyourscript(atleastforawhile)sothatthesearenotmutuallyexclusivetechniques.

UsingthebreakpointresourceThebreakpointresourceisstructuredjustlikeanyotherChefresource.Theresource’snameattributeisthelocationwhereyouwanttoinsertthebreakpoint,andithasonlyoneaction,:break,whichsignalschef-shelltointerruptexecutionofthecurrentrecipeandprovideaninteractiveshell.Anybreakpointresourcesinrecipesareignoredbythechef-client.Thatway,iftheyareforgottenaboutandleftinarecipe,theywillnotcausehavocinproduction.Thatbeingsaid,theyshouldonlybeusedwhenactivelydebugginganissueandremovedbeforereleasingyourrecipesintoyourproductionenvironment.

Thenameattributehasthefollowingstructure:

whenresourceresource_name

Here,whenhasthevalueof“before”or“after”,toindicatewhetherthebreakpointshouldstopbeforeorafterexecution,respectivelyandresourceisthetypeofresourcethatwhencombinedwithresource_nameistheuniqueidentifierthatwilltriggerthebreakpoint.Forexample:

beforefile'/tmp/foo.txt'

Thiswouldcausetheshelltointerruptexecutionoftherecipesimmediatelybeforeanyfileresourcethatwasmanipulating/tmp/foo.txt.Anotherexample,wherewewanttostopexecutionafterinstallingthegitpackage,wouldlooklikethefollowing:

afterpackage'git'

Usingthis,wewilltellchef-shellthatexecutionwastobepausedoncethegitpackagewasmodified.Let’slookathowwecanformasimplerecipecompletewithbreakpointresourcesthatwouldusetheseexamples:

breakpoint"beforefile'/tmp/foo.txt'"do

action:break

end

breakpoint"afterpackage'git'"do

action:break

end

file'/tmp/foo.txt'do

action:create

end

package'git'do

action:remove

end

Forthosewhohaveusedgdboranyotherdebugger,thiswillbeeasytounderstand;ifyouhavenotusedaninteractivedebugger,thentryafewoftheinteractiveexamples,andyouwillgetthehangofitinnotimeatall.

Chefshellprovidesacomprehensivewaytointeractwithyourrecipes.Nowthatyouseehowtotestoutanddebugyourwork,let’stakealookathowwecangoonestepfurtherinourtestingtoperformfullend-to-endintegrationtestingofourinfrastructure.

IntegrationtestingWithintegrationtesting,yourtestsmoveintotestingbeyondyourcode.WithChefSpec,welookedathowtoverifythatarecipewasexecutingstatementsasexpected.However,unittestinghasitslimits;integrationtestingcompletesthepicturebytestingcookbooksinconjunctionwithrealhoststhatrunthedesiredoperatingsystem(s):

Thismeanstestingcookbooksfromtheoutsideratherthanfromtheinside,whereaunittestprovidesverynarrowtestingofcodeinsideofcookbooks,integrationtestsperformdeploymentofyourcookbooksinatest-specificenvironmentandthenexecuteprobestovalidatethatthesystemisinthedesiredstate.OneofthemostpopulartoolsforthisisTestKitchen,whichwewilltakeabrieftourof.

UsingTestKitchenTestKitchenisanopensourcetoolthathelpstoautomateintegrationtestingofChefcookbooks.WhereaChefSpectestwouldvalidatethatafileresourcewascalledduringanexecution,asimilarlytaskedTestKitchentestwouldspinupanewinstance,executeyourrecipe(s),andthenvalidatethatthefilewasactuallycreatedonallplatformsthatyouexpecttherecipetoworkon.OneofthegreatthingsaboutTestKitchenisthatitsupportsanumberofdifferentdriverpluginstomanageyourtargethosts,includingVagrant(thedefault),EC2,OpenStack,Docker,andRackspaceCloudamongothers.ThisenablesyoutotestyourcookbooksnotonlyonlocalvirtualmachinesusingVagrant,buttoverifythattheyworkcorrectlyonacloudserviceaswell.Theabilitytoperformintegrationtestsondifferenttypesofhostsbringsyourteststhatmuchclosertomatchingaproductionenvironment,therebyincreasingyourconfidenceinthechangesyouaremaking.

InstallingTestKitchenCurrently,TestKitchenisdistributedasaRubygem,soinstallationisquitestraightforward:

geminstalltest-kitchen

Byinstallingthisgem,youarealsoinstallingacommand-linetool,kitchen,whichisusedtointeractwithTestKitchen.SimilartoVagrant,Bundler,andothertools,TestKitchenusesaconfigurationfiletostoreinformationaboutwhattotest,wheretotestit(onwhatvirtualmachines),andhowtotestit.

TestingwithTestKitchenAsmentionedbefore,TestKitchenfocusesonintegrationtestingofyourChefcomponents.Thismeansthatitneedstobeabletoexecuteyourrecipesonahost(orasetofhosts)andtheninvokeasetofteststovalidatethattheexpectedbehavioroccurredontheendhost(s).TestKitchenisresponsibleforthefollowing:

ProvisioningacleanhostfortestingInstallingChefontothenewhostExecutingourrecipesonthehostValidatingthebehavioroftherecipesonthehost

So,let’sgetstartedwithsomeactualtesting!

Buildingasimplecookbook

InordertodemonstratehowtouseTestKitchen,wewillneedtowriteacookbookthatwecanrunonourtesthostsandwriteteststovalidatethebehavior.InordertofocusonhowtouseTestKitchen,wewilltakealookataverysimplecookbookthatjustcreatesafileonthehostfilesystemsothatitisguaranteedthatitwillworkonbothUbuntuandCentOSplatforms.

First,createaplacetodoyourwork:

mkdir-ptestfile-cookbook/recipes

Then,addaverysimplemetadata.rbinyourtestfile-cookbookdirectorywiththefollowingcontentsinside:

name"testfile"

version"1.0"

Oncethatiscomplete,addadefaultrecipe,recipes/default.rb,withthesamplerecipecodeasfollows:

file"/tmp/myfile.txt"do

content"Myawesomefile!"

end

Nowthatwehaveacomplete(albeitsimple)cookbook,let’stakealookathowtotestitusingTestKitchen.

Preparingyourcookbookforthekitchen

InordertostartusingTestKitchen,youwillneedtoprepareyourtestenvironment.TestKitchenreliesonaconfigurationfile,.kitchen.yml,totellitwhattodo.Youcangenerateitbyhand,oryoucouldusetheinitcommandaspartofthekitchentool:

kitcheninit

Thiswilldoafewthingsforyou:first,itwillcreatea.kitchen.ymlfilewithsomesanedefaultsinthecurrentdirectory.Then,itwillcreateatest/integration/defaultdirectoryforyourintegrationtests,andthenitwillinstalltheVagrantdriverforTestKitchensothatitcaninteractwithVagrantvirtualmachines(ifithasnotalreadybeeninstalled).

Ifyoulookatthe.kitchen.ymlfile,youwillseethattheinitialfilecontainsthefollowingYAMLcode:

---

driver:

name:vagrant

provisioner:

name:chef_solo

platforms:

-name:ubuntu-12.04

-name:centos-6.4

suites:

-name:default

run_list:

-recipe[testfile::default]

attributes:

ThisconfigurationfileinstructsTestKitchentouseVagranttomanagethetargetinstancesandtouseChef-soloforprovisioning,anditshouldexecutethedefaultsuiteoftestsonbothUbuntu12.04andCentOS6.4.Ofcourse,youcanalwaysmodifyorextendthislistifyouhaveothersystemsthatyouwanttoruntestson,butthisisareasonabledefaultlist

fornow.

Noticethatwedon’thaveanyattributesspecifiedasourdefaultrecipeanditdoesnotuseattributes.Ifyouneedtoprovideattributestotestrecipes,thiswouldbetheplacetodoit,whichislaidoutasadictionaryinYAML.Eachtestsuitehasitsownrunlistanddefinedattributesthatallowyoutocheckthebehaviorofavarietyofconfigurationdataandrecipecombinations.

Testingyournewcookbook

TestingacookbookwithTestKitchenisoutlinedinthefollowingthreesteps:

1. Provisioningahostifneeded.2. Convergingthehostsothatitisuptodate.3. Executingtests.

Let’stakealookathowwewillperformthesewithoursimplecookbooktocheckthatitworksproperly.Provisioningtheinstance

Beforeyoucantest,youwillneedtopreparetheinstancesfortesting—thisisdoneusingthekitchencreate<instancename>command.Onlyyoudon’tknowwhatinstancetobringupjustyet.Togetthelistofinstancesthatcanberun,wewillusethelistsubcommand:

[jewart]$kitchenlist

InstanceDriverProvisionerLastAction

default-ubuntu-1204VagrantChefSolo<NotCreated>

default-centos-64VagrantChefSolo<NotCreated>

Youwillseethatthislistisgeneratedbyacombinationoftheplatformsandthesuiteslistedinthe.kitchen.ymlfile.Ifyouweretodefineanewsuite,namedserver,thenyourlistwouldincludetwoadditionalinstances,server-ubuntu-1204andserver-centos-64.

Onceyouhaveseenthislist,youcancreateanUbuntu12.04instancewiththefollowingcommand:

kitchencreatedefault-ubuntu-1204

ThiswilluseVagrantandVirtualBoxtoprovisionanewheadlessUbuntu12.04hostandbootitupforyoutostarttestingwith.Ifyoudon’talreadyhaveanUbuntu12.04Vagrantimagedownloaded,thenitwillbedownloadedforyouautomatically(thisisalargeimagesoitmaytakeawhiletocompletethisoperation,dependingonyourconnectionspeed,ifyouarefollowingalong).

ThiswilllookfamiliartothosethatusedVagrantatthebeginningofthechapter:

[jewart]%kitchencreatedefault-ubuntu-1204

----->StartingKitchen(v1.2.1)

----->Creating<default-ubuntu-1204>...

Provisioninghappens…

Finishedcreating<default-ubuntu-1204>(4m2.59s).

----->Kitchenisfinished.(4m2.83s)

However,thisonlybuildsthevirtualmachineforourtests;itdoesnotrunourrecipesorourtestsinthenewlyconstructedhost.Beforewemoveontowritingatest,let’stakealookathowtorunourrecipeinsideourinstance.Convergingthenewlycreatedinstance

Thenextstepistoexecuteourrunlist(“converge”inChefparlance)onthenewinstance.ThisisdonewithTestKitchen’sconvergecommandandcanbeusedforalloronespecificinstance.InordertoconvergeourUbuntu12.04instance,thefollowingcommandisused:

kitchenconvergedefault-ubuntu-1204

Whatthiswilldoistransferanyrequireddatafiles,installChefasneeded,andthenexecutetherunlistforthespecifiedsuite(defaultinthiscase)ontheinstance.Hereisasamplerunofconverge(withsomepartsremoved):

[jewart]%kitchenconvergedefault-ubuntu-1204

----->StartingKitchen(v1.2.1)

----->Converging<default-ubuntu-1204>...

Preparingfilesfortransfer

Preparingcurrentprojectdirectoryasacookbook

Removingnon-cookbookfilesbeforetransfer

----->InstallingChefOmnibus(true)

InstallationofChefhappens…

CompilingCookbooks…

Converging1resources

Recipe:testfile::default

AndthenyouwillseetheoutputofanormalChefrunafterthat—atthispoint,therunlistiscompleteandtheinstancehasbeenconvergedtothelateststate.Nowthatit’sbeenconverged,wewillcontinuetowriteaverysimpletesttoverifythatourrecipedidtherightthing.Writingasimpletest

TestsarestoredinadirectorywhosestructureissimilartootherChefcomponents—thedirectorywecreatedpreviouslywithkitcheninit,tests/integration/default,allowsustokeepourintegrationtestsseparatefromspectestsorothertypesoftests.Theintegrationdirectorywillcontainonedirectorypersuitesothattestfilesaregroupedtogetherbasedontheparticularcomponentoraspectofyourcookbookthatisbeingtested.Additionally,dependingonthetypeofthetestframeworkbeingused,yourtestswillbecontainedinanotherchilddirectoryforthegivensuite.Inthiscase,wewilltakealookattheBASHAutomatedTestingSystem(BATS),soourtestwillbeplacedinthetests/integration/default/bats/file_created.batsfileandwilllooklikethefollowingcode:

#!/usr/bin/envbats

@test"myfile.txtexistsin/tmp"{

[-f"/tmp/myfile.txt"]

}

Thisallowsustousethesimple-fBASHtest(whichreturnstrueifthespecifiedvalueexists)toguaranteethatthefilewascreatedontheinstance.

Next,wecanrunthistestwithkitchenverifydefault-ubuntu-1204andseethattheBATSpluginwasinstalledandthatourtestwasexecutedandpassed:

[jewart]%kitchenverifydefault-ubuntu-1204

----->StartingKitchen(v1.2.1)

----->Settingup<default-ubuntu-1204>...

Fetching:thor-0.19.0.gem(100%)

Fetching:busser-0.6.2.gem(100%)

Successfullyinstalledthor-0.19.0

Successfullyinstalledbusser-0.6.2

2gemsinstalled

----->SettingupBusser

CreatingBUSSER_ROOTin/tmp/busser

Creatingbusserbinstub

Pluginbatsinstalled(version0.2.0)

----->Runningpostinstallforbatsplugin

InstalledBatsto/tmp/busser/vendor/bats/bin/bats

Finishedsettingup<default-ubuntu-1204>(0m8.94s).

----->Verifying<default-ubuntu-1204>...

Suitepathdirectory/tmp/busser/suitesdoesnotexist,skipping.

Uploading/tmp/busser/suites/bats/file_created.bats(mode=0644)

----->Runningbatstestsuite

√myfile.txtexistsin/tmp

1test,0failures

Finishedverifying<default-ubuntu-1204>(0m0.67s).

----->Kitchenisfinished.(0m9.86s)

Todemonstratewhathappensifthefiledoesnotexist,wecancloneourtesttocreateasecondsimpletestthatvalidatestheexistenceof/tmp/myotherfile.txt,andrunourverifycommandagainwithoutmakingacorrespondingchangetoourrecipe.TheoutputfromTestKitchenwilltellusthatourtestfailedandwhy:

----->Runningbatstestsuite

√myfile.txtexistsin/tmp

x

myotherfile.txtexistsin/tmp

(intestfile/tmp/busser/suites/bats/other_file.bats,line4)

2tests,1failure

Command[/tmp/busser/vendor/bats/bin/bats/tmp/busser/suites/bats]exit

codewas1

>>>>>>Verifyfailedoninstance<default-ubuntu-1204>.

Combiningallthesteps

Fortunately,thefinefolksthatcreatedTestKitchenrealizedthatitwouldbetedioustorunallthreestepseverytimeyouwantedtorunsometests.Asaresult,thereisthekitchentestcommandthatwillprovisionaninstance,executetherunlist,verifytheresults,andthenteardowntheinstancewithonlyonecommand.Inthiscase,youcanreplacethemwiththefollowingsinglecommand:

kitchentestdefault-ubuntu-1204

ThiscoversthebasicsoftestingyourcookbookswithTestKitchen.ThereareotherthingsthatcanbedonewithTestKitchen,includingusingothertestingmechanisms,testingcookbookdependencies,validatingwhetherservicesarerunning,fullyautomatingTestKitchenaspartofyourreleaseprocess,andplentymore.Formoreinformation,visittheprojectathttp://kitchen.ci/.

ExtendingChefChefisdevelopedintheopenwithflexibilityandextensibilityinmind.Mostofthetoolsarearchitectedtosupportloadingcustompluginstosupportdevelopmentofadd-onsfornewfunctionality.Asyousawearlier,Knife’scloudservicesupportisprovidedbyplugins,oneforeachcloudservice,includingEC2,Azure,andRackspacecloud.WewilllookathowthathappenssothatyoucanexplorewritingyourownpluginsforKnife,Ohai,andotherChefcomponentsshouldtheneedarise.

InadditiontoextendingChef’scorecomponentsdirectly,itispossibletoextendthefunctionalityofyourChefecosystembybuildingenhancementstoexistingtoolsthatcanleverageChef’sdataAPIstoprovidedataaboutyourinfrastructure.

WritinganOhaipluginOhai(aplayonthephraseoh,hi!)isatoolthatisusedtodetectattributesonanode,andprovidetheseattributestothechef-clientatthestartofeverychef-clientrun.WithoutOhai,thechef-clientwillnotfunction,andtherefore,itmustbepresentonanodeinorderforCheftowork.Thedatathatiscollectedisauthoritative—ithasthehighestlevelofprecedencewhencomputingattributedataforclientrunsonanode.

ThetypesofattributesthatOhaimightbeusedforinclude:

PlatformdetailsNetworkusageMemoryusageProcessorusageKerneldataHostnamesInformationaboutthenetworktopologyCloud-specificinformation

Ohaiimplementsanextensiblearchitecturethroughpluginsthatallowsenduserstowritecustomextensionstoreportinformationthatiscollectedaboutanode.Forexample,therearepluginsforEC2cloudhoststhatuseEC2-specificmechanismstodetermineinformationaboutthehost,includingitsinternalIPaddressandotherbitsofinformation.

ThisisincrediblyusefulforintegratingChefwithanexistinginfrastructure,asyoucanautomaticallyprobeforlocalconfigurationdataandgenerateattributesfromthatdata.OncethisdataisstoredinChef,itcanbeusedinsearchqueries,recipes,andeverywhereelsethatyoucouldotherwiseusenodeattributes.

EveryOhaipluginfollowsapattern:itregistersitselfasprovidingacertainclassofattributedataandcontainsacombinationofgeneralpurposeanddatacollectionmethodstogatherinformationaboutthelocalstateofthesystemandreportitbacktotheChefserver.Ohaialreadyhasbuilt-incollectorsforanumberofplatforms,includingLinux,Windows,andBSD.

AsampleOhaipluginwouldlooklikethefollowing:

Ohai.plugin(:Region)do

provides"region"

definit

regionMash.new

end

collect_data(:default)do

#Runsonallhostswhoseplatformisnotspecificallyhandled

init

region[:name]="unknown"

end

collect_data(:linux,:freebsd)do

#RunonlyonLinuxandFreeBSDhosts

init

region[:name]=discover_region_unix

end

collect_data(:windows)do

#RunonWindowshosts

init

region[:public_ip]=discover_region_windows

end

end

Hereweregisterourpluginasprovidingaregiondata,thereisaninitmethodthatcreatesaMashforourregiondata,andafewdatacollectioncallbacks.Eachdatacollectioncallbackisregisteredasablockthatiscalledforthespecifiedplatform(s).Inourcase,therearethreecallbacksregistered,oneforWindowssystems,oneforLinuxandFreeBSDsystems,andthenafallbackthatwillbecalledforanyplatformnotexplicitlyhandled.

AnoteaboutwritingOhaipluginsThewaywedeclaretheregionMashinourinitmethodisalittlebitdifferentthannormalvariableassignmentinRuby.Inourplugin,wedefinetheplugin’sregionpropertywiththefollowingcode:

regionMash.new

Onthesurface,thismightlooklikesomeoneforgotanequalssign,asinthefollowingRubycode:

region=Mash.new

However,iftherewereanequalssign,thepluginwouldnotworkasintended.Inthiscase,thereisnoequalssignmissingandthecodeiscorrect.ThisisbecauseOhai’sPluginclassleveragesaspecialRubymechanismforinterceptingcallstononexistentmethodsanddynamicallyhandlingthem.Thismechanismisrecognizablebythepresenceofaspecialmethodnamedmethod_missinginthecode.Inthiscase,themethod_missinghandlerwillcallaspecialOhaipluginmethod,get_attribute,ifnoargumentsarepassed,oritwillcalltheset_attributemethodifargumentsarepassed.

Todemonstratewhythisisused,ifyouwantedtohavethesameeffectwithoutthemethod_missingmechanism,thentheplugin’sinitmethodcouldbewrittenas:

definit

set_attribute"region",Mash.new

end

Wereyoutodothis,thenoursubsequentcollectdatamethodswouldneedtoberewrittenaswell.Hereisanexampleofwhattheymightlooklike:

collect_data(:platform)do

init

region=get_attribute"region"

region[:name]=get_data_mechanism

set_attribute"region",region

end

Youcanseethatthemethod_missingmechanismmakeswritingpluginsmorenatural,thoughittakesalittlebitofextraworktounderstandhowtowritethematfirst.

ChefwithCapistranoOneexampleofextendingChefthroughexternaltoolsisthecapistrano-chefgemthatextendsthepopulardeploymenttool,Capistrano.WritteninRuby,Capistranowasdesignedtodeployapplicationsandperformlightsystemsadministration.IfyouhaveexistingapplicationsthatarebeingdeployedusingCapistrano,thisisanexampleofhowtoleverageyourconfigurationdatastoredinCheftomakeintegrationasseamlessaspossible.

IfyouhaveanexistingapplicationthatusesCapistrano,youwillhaveadeploy.rbfilethatdefinesthevariousapplicationroles.EachtoolhasanarrayofIPaddressestohoststhatprovidethatroleandmightlooksomethinglikethis:

role:web,'10.0.0.2','10.0.0.3'

role:db,'10.0.0.2',:primary=>true

Byusingcapistrano-chef,youcandothis:

require'capistrano/chef'

chef_role:web,'roles:web'

chef_role:db,'roles:database_master',

:primary=>true,

:attribute=>:private_ip,

:limit=>1

NoticethatherewehaveusedasimplesearchquerytodeterminethehoststhatshouldbeincludedineachCapistranorole.Inthiscase,the:webrolehasbeenreplacedwithaChefsearchqueryforallnodesthathavethewebChefroleassociatedwiththem.ThisallowsyoutomodelyourdatainChef,butstilluseCapistranotodeployyourapplicationstack,increasingtheeaseofintegration.

ThisworksbecauseallofChef’sdataisavailableviaanHTTPAPImakingintegrationassimpleasmakingHTTPcallsandparsingsomeJSONresults(which,comparedtosomeotherintegrationmechanisms,isincrediblyeasy).

AutomationandintegrationOneofthebestpartsaboutChefisthatyourinfrastructureandsoftwaremodelisconsistentwithwhatisdeployed.Whatthismeansforyouisthatwhenchef-clientrunsonanendhost,thathost’sstateisupdatedtomatchyourmodeledenvironment.Forexample,considerascenariowhereyouhave10EC2databasehosts,andallofthemhaveaspecialrole,database_server,appliedtothem.Thisrole’sattributesindicatethatPostgreSQL9.1istobeinstalledanditsdatashouldbestoredin/opt/postgresql/data.Byexecutingchef-clientonalltennodes,theywillhavePostgreSQL9.1installedandstoringdatain/opt/postgresql/data.NowconsiderthatallofournodesneedtohaveanewEBSstoragedeviceattachedtoeachofthem,andPostgreSQLneedstobepointedtoournewEBSdevice.UpdatingourmodeltoincludearecipethatmountstheEBSdevicegracefullyshutsdownPostgreSQL,movesthedatabasedata,reconfiguresPostgreSQL,andstartsitupagain.Wecanautomateandrolloutthisconfigurationtoourfleetoftendatabasehosts.Youcaneasilyimaginetenhostsgrowingtohundredsoreventhousands.Thisiswhatthepowerofautomationisallabout.

AutomatedupdatesanddeploymentsIfyouhaveconfidenceinyourmodelandyourcookbooks,thenyoucantakethisautomationonestepfurther.Byautomatingtheexecutionofchef-clientonaperiodicschedule,youcanfullyautomateupdateswithoutneedingtoSSHintothehoststorunchef-client.However,thislevelofautomationrequiresahighlevelofconfidenceinthecorrectnessofyourcookbooks.Achievingthisrequirescontinuousandin-depthtestingofnotonlythecodeinthecookbooksdevelopedbutalsoofthedependenciesthatareneededtomakeyourcookbookswork.Tothatend,comprehensiveintegrationtestscanhelptobuildtheconfidenceneededtomoveintoafullyautomatedworld.

SummaryBynowyouhavebeenexposedtoalotofwhatChefhastooffertheDevOpscommunity.YouhaveseenwhatChefdoes,howtoinstallit,andhowitworks.Throughoutthisbook,youhavebeenintroducedtosomenewwaysofthinkingabouthowtomodelinfrastructureanduseautomatedtoolstomanageit.

Atthispoint,youhopefullyunderstandhowtomodelyourinfrastructurewithChefaswellasinstallthevariouscomponentsrelatedtoChef,rangingfromtheservertotheclient.Fromhere,youcantakewhatyou’velearnedaboutthevariouscomponentsofChefandusethatinformationtobuildmoreadvancedcookbookstodeployyoursoftwareandmanageyourinfrastructure,rangingfromcloudhoststophysicalon-sitehardwareandevenvirtualmachinesusingVagrant.Onceyouhavegottenthingsworking,youcanautomateyourconfigurationtoolsandensurethereliabilityofyourcookbooksthroughunitandintegrationtestsaswell.

Now,itisyourturntotakethisinformationandyournewskillstoautomateyoursystemsinfrastructureinordertobuildexcitingnewthings!

IndexA

AmazonEC2about/AmazonEC2workingwith/AmazonEC2knifeplugin,installing/InstallingtheEC2knifepluginauthentication,settingup/SettingupEC2authenticationinstance,provisioning/Provisioninganinstanceinstance,bootstrapping/Bootstrappingtheinstanceinstance,terminating/TerminatingtheinstanceChefnode,removing/RemovingtheChefnode

AmazonEC2authenticationsettingup/SettingupEC2authentication

AMIIDURL/Provisioninganinstance

ApacheSolrdocumentationURL/Queryingyourdata

applicationconfiguring/Configuringyourapplication

applicationdeploymentcookbookviewing/Lookingatyourapplicationdeploymentcookbook

applicationrunmaintaining/Keepingyourapplicationrunning

applicationtemplatedefining/Definingafullapplicationtemplate

attributes/GettingtoknowChefabout/Attributesconfigurationvalues/Attributesmultiplefiles/Multipleattributefilesusing/Usingattributes

authorizedkeys,SSHkeystemplating/Templatingtheauthorizedkeys

automation,Chef/Automationandintegration

BBase64encoding/Secretkeysbaselinerole/ModelingasimplePythonapplicationbaseserverrole

creating/CreatingthebaseserverroleBASHAutomatedTestingSystem(BATS)/Writingasimpletestbootstrap/GettingtoknowChefbootstrapscript/GettingtoknowChefbreakpointresource,Chefshell

using/UsingthebreakpointresourceBundler,ChefSpec/InstallingChefSpec

CCapistran

Chef,usingwith/ChefwithCapistranoChef

terminology/Terminologyworkingwith/WorkingwithChefbenefits/WorkingwithChef,Large-scaleinfrastructurechef-solo,installing/Installingchef-soloRubygemmechanism/TheRubygemgems,managing/Managinggemschef-solo,verifying/Verifyingthatchef-soloworksabout/GettingtoknowChefnode/GettingtoknowChefworkstation/GettingtoknowChefbootstrap/GettingtoknowChefbootstrapscript/GettingtoknowChefrecipe/GettingtoknowChefcookbook/GettingtoknowChefattributes/GettingtoknowChefrole/GettingtoknowChefrunList/GettingtoknowChefresource/GettingtoknowChefprovider/GettingtoknowChefdatabags/GettingtoknowChefenvironment/GettingtoknowChefused,forsoftwaredeployment/DeployingsoftwarewithChefdependencies,managing/ManagingdependenciesinChefextending/ExtendingChefOhaiplugin,writing/WritinganOhaipluginusing,withCapistran/ChefwithCapistranoautomation/Automationandintegrationintegration/Automationandintegrationupdates,automating/Automatedupdatesanddeployments

chef-server-ctl,Chefserverworking/Understandinghowchef-server-ctlworks

Chef-soloabout/VagrantandChef-soloVagrant,combiningwith/CombiningVagrantwithChef-solo,ConfiguringChef-solo,TellingChef-solowhattorunlimitations/UnderstandingthelimitationsofChef-soloconfiguring/ConfiguringChef-solocookbooks_path/ConfiguringChef-solodata_bags_path/ConfiguringChef-solo

roles_path/ConfiguringChef-solorundirections,providing/TellingChef-solowhattorunroles,usingwith/UsingrolesanddatabagswithChef-solodatabags,usingwith/UsingrolesanddatabagswithChef-solo

chef-soloinstalling/Installingchef-soloverifying/Verifyingthatchef-soloworks

ChefcommunityURL/Managingthecookbooks

ChefLWRP,componentresource/Buildingaresourceprovider/Buildingaresource

Chefnode,AmazonEC2removing/RemovingtheChefnode

Chefnode,RackspaceCloudremoving/RemovingtheChefnode

Chefserverinstalling/InstallingaChefserverURL/Gettingtheinstallerconfiguring/ConfiguringaChefserverinteracting,Chefshellused/InteractingwiththeChefserverusingtheshell

Chefserver,installingonRedHatEnterpriseLinuxpackage,downloading/Downloadingthepackagepackage,downloading,URL/Downloadingthepackage

Chefserver,installingonUbuntuabout/InstallingonUbuntupackage,downloading/Downloadingthepackagepackage,downloading,URL/Downloadingthepackagepackage,installing/Installingthepackage

Chefserverconfigurationabout/ConfiguringaChefserverchef-server-ctl,working/Understandinghowchef-server-ctlworkshostserveractions,monitoring/What’shappeningonmyserver?services,verifying/Verifyingthattheservicesarerunning

Chefserverinteraction,withChefshelldata,interactingwith/Interactingwithdatadata,searching/Searchingyourdatadata,editing/Editingyourdatadata,transforming/Transformingdata

Chefserviceabout/Terminologyvalidating/Validatingthatyourserviceisworkingknifeconfiguration/Ensuringthatyourknifeconfigurationworks

Chefshell

about/GettingtoknowtheChefshellusing/UsingtheChefshellstandalonemode/Thestandalonemodesolomode/Thesolomodeclientmode/Theclientmodeused,forinteractingwithChefserver/InteractingwiththeChefserverusingtheshellused,forexecutingrecipes/ExecutingrecipeswithChefshellrecipe,creatingin/Creatingarecipeintheshellused,fordebugging/DebuggingwiththeChefshellbreakpointresource,using/Usingthebreakpointresource

ChefSpecabout/TestingrecipesandRSpec/RSpecandChefSpectestingbasics/Testingbasicsusing/UsingChefSpecoverview/GettingstartedwithChefSpecinstalling/InstallingChefSpecdependencies,lockinginRuby/LockingyourdependenciesinRubysimplerecipe,creating/CreatingasimplerecipeandamatchingChefSpectest

ChefSpectestwriting/WritingaChefSpectestexpanding/Expandingyourtestsmultipleexamples/Multipleexamplesinaspectesttesting,formultipleplatforms/Testingformultipleplatforms

ChefSupermarketURL/Determiningwhichrecipesyouneed

clientmode,Chefshell/Theclientmodecloud

leveraging/Leveragingthecloudcloudplatformproviders

AmazonEC2/AmazonEC2RackspaceCloud/RackspaceCloud

components,forChefserverinstallation/Whatyouwillbeinstallingconfigurationblocks

using/Usingconfigurationblocksconfigurationdata

organizing/Organizingyourconfigurationdatasources/Organizingyourconfigurationdataattributedataexample/Exampleattributedatadatabags/Databags

cookbook/Terminologyabout/GettingtoknowChef,Installingacookbookinstalling/Installingacookbook

cookbook,TestKitcheninstance,provisioning/Provisioningtheinstancecreatedinstance,converging/Convergingthenewlycreatedinstancetest,writing/Writingasimpletestsummarizing/Combiningallthesteps

cookbooksmanaging/ManagingthecookbooksURL/Managingthecookbooksdownloading/DownloadingcookbooksURL,fordownloading/Downloadingcookbooksdatabaserecipe,viewing/Lookingatthedatabaserecipeapplicationdeploymentcookbook,viewing/Lookingatyourapplicationdeploymentcookbookdirectories,preparing/PreparingthedirectoriesPythonvirtualenvironment,constructing/ConstructingyourPythonvirtualenvironmentsourcecode,checkingout/Checkingthesourcecodedependencies,installing/InstallinganyextradependenciesPython’srequirementsfile,using/UsingPython’srequirementsfileapplication,configuring/Configuringyourapplicationapplicationrun,maintaining/Keepingyourapplicationrunning

customdefinition,developingabout/Developingacustomdefinitioncode,organizing/Organizingyourcodedefinition,writingforusingPIP/WritingadefinitionforusingPIPapplicationtemplate,defining/Definingafullapplicationtemplate

customextensionswriting/Writingcustomextensionsresource,building/Buildingaresource

Ddata

storing,indatabags/Storingdataindatabagssearching/Searchingfordata,Searchingyourdataencrypting/Encryptingyourdatadecrypting/Decryptingyourdataquerying/Queryingyourdata

data,Chefserverinteracting/Interactingwithdatasearching/Searchingyourdataediting/Editingyourdatatransforming/Transformingdata

data,searchingdatabags,searchingwithknifetool/Searchingyourdatabagswithknifedatabags,searchingfromrecipes/Searchingyourdatabagsfromarecipedata,querying/Queryingyourdata

databagsabout/GettingtoknowChef,Databagsusing/Knowingwhentousedatabagsdata,storingin/Storingdataindatabagscreating,forusers/Creatingadatabagforusersworkingwith/Workingwithdatabagssecuring/Securingyourdatabagsdata,searching/Searchingyourdatabagswithknifemultiplemachines,managingwithsearchqueries/Managingmultiplemachineswithsearchqueries

databagssearching,withknifetool/Searchingyourdatabagswithknifesearching,fromrecipes/Searchingyourdatabagsfromarecipe

databags,securingabout/Securingyourdatabagssecretkeys/Secretkeysdata,encrypting/Encryptingyourdatadata,decrypting/Decryptingyourdatakeys,storingonnodes/Storingkeysonnodes

databags,usingwithChef-soloabout/UsingrolesanddatabagswithChef-solocustomJSONdata,injecting/InjectingcustomJSONdatacustomnodename,providing/Providingacustomnodename

databasehostconfiguring/Configuringthedatabasehost

databaserecipeviewing/Lookingatthedatabaserecipe

databaseserverrolecreating/Creatingthedatabaseserverrole

defaultaction/Overridingadefaultbehaviordefaultbehavior,resource

overriding/Overridingadefaultbehaviordefinition/Developingacustomdefinitiondefinitions

about/Definitionsdependencies

installing/Installinganyextradependenciesmanaging,inChef/ManagingdependenciesinChefmanaging/Managingdependencieselsewhere

deploymentkeys,SSHkeysadding/Addingdeploymentkeys

deployusers/Addingdeploymentkeysdirectories

preparing/Preparingthedirectoriesdriverplugins/UsingTestKitchendry-runmode/Implementingtheprovider

EEC2instances

provisioning/ProvisioningEC2instancesdatabasehost,configuring/Configuringthedatabasehost

echooffstate/Creatingarecipeintheshellechoonstate/Creatingarecipeintheshellencryptionkey

storing,onnodes/Storingkeysonnodesenvironment

about/GettingtoknowChefexample/Environments

equalsign/VariablereplacementERBprimer

about/AquickERBprimerURL/AquickERBprimerRubycode,executing/ExecutingRubyRubycodeexecution,variablereplacement/Variablereplacement

Ffetch_from_s3method/Implementingtheprovider

Ggems

managing,withRVM/Managinggemsgemset/Managinggems

HHostedChef

URL/WorkingwithChef

IImage-O-Rama/Determiningwhichrecipesyouneedimage-processingrole

defining/Animage-processingroleimagesearchrole

defining/Animagesearchroleinfrastructuremodeling

steps/Modelingyourinfrastructureabout/ModelingyourinfrastructureChef,using/Modelingyourinfrastructureservice-orientedwebapplication,services/Modelingyourinfrastructureservice-orientedarchitecture,advantages/Modelingyourinfrastructurerole/Rolesrole,implementing/Implementingarolerecipes,determining/Determiningwhichrecipesyouneedrecipes,applyingtorole/Applyingrecipestorolesrole,mappingtonode/Mappingyourrolestonodesrole,mappingtonodes/Mappingyourrolestonodesnodes,converging/Converginganodeenvironment/Environments

installationVagrant/InstallingVagrant

installation,Chefserverabout/InstallingaChefserverrequisites/Requirementsandrecentchanges,Installationrequirementscomponents/Whatyouwillbeinstallingomnibusinstaller,obtaining/GettingtheinstalleronUbuntu/InstallingonUbuntuonRedHatEnterpriseLinux/InstallingonRedHatEnterpriseLinux

instance,AmazonEC2provisioning/Provisioninganinstancebootstrapping/Bootstrappingtheinstanceterminating/Terminatingtheinstance

instance,RackspaceCloudprovisioning/Provisioninganinstanceterminating/Terminatinganinstance

integration,Chef/Automationandintegrationintegrationtesting

about/IntegrationtestingTestKitchen,using/UsingTestKitchen

Kknife-rackspaceplugin/Provisioninganinstanceknifenodelist/InteractingwiththeChefserverusingtheshellknifeplugin,AmazonEC2

installing/InstallingtheEC2knifepluginknifesearchnode…/InteractingwiththeChefserverusingtheshellknifetool/Validatingthatyourserviceisworking

used,forsearchingdatabags/Searchingyourdatabagswithknife

Llarge-scaleinfrastructure/Large-scaleinfrastructureload_current_resourcemethod/Loadinganexistingresourcelocalenvironment

configuring/Configuringyourlocalenvironment

MMash,Chef/UsingattributesmatchingChefSpectest

creating/CreatingasimplerecipeandamatchingChefSpectestmetadata

about/Metadatamockmethods/RSpecandChefSpecmultipleattributefiles

about/Multipleattributefilesmultipleplatforms,supporting/Supportingmultipleplatformsexternalattributes,loading/Loadingexternalattributes

multiplemachinesmanaging,withsearchqueries/Managingmultiplemachineswithsearchqueries

mysql/Lookingatthedatabaserecipe

N*nameparameter/WritingadefinitionforusingPIPnewhost

provisioning,withVagrant/ProvisioninganewhostwithVagrantnode/Terminology

about/GettingtoknowChefrole,mappingto/Mappingyourrolestonodesconverging/Converginganode

nodeattributesdefining/Definingnodeattributessetting/Settingattributesaccessing/Accessingattributes

nodedeletecommand/RemovingtheChefnodenodehash/Usingattributes

OOhaiplugin

writing/WritinganOhaiplugin,AnoteaboutwritingOhaipluginsattributes/WritinganOhaiplugin

omnibusinstallationpackage,Chef/Requirementsandrecentchangesomnibusinstaller,Chefserver

obtaining/Gettingtheinstalleroutline/Installationoutlinesteps/Installationoutline

OpenSSHserviceroledefining/AnOpenSSHservicerole

PPIP

used,forwritingdefinition/WritingadefinitionforusingPIPpostgresql/LookingatthedatabaserecipePostgreSQLservicerole

defining/APostgreSQLserviceroleprovider/GettingtoknowChef

about/Buildingaresourceimplementing/Implementingtheprovider

Pythonapplicationmodelling/ModelingasimplePythonapplication

Pythonrequirementsfileusing/UsingPython’srequirementsfile

Pythonvirtualenvironmentconstructing/ConstructingyourPythonvirtualenvironment

Python’srequirementsfileusing/UsingPython’srequirementsfile

RRackspaceCloud

about/RackspaceCloudinstance,provisioning/Provisioninganinstanceinstance,terminating/TerminatinganinstanceChefnode,removing/RemovingtheChefnode

read-eval-print-loop(REPL)/GettingtoknowtheChefshellrecipe/Terminology

about/GettingtoknowChefdetermining/Determiningwhichrecipesyouneedneedfor/Determiningwhichrecipesyouneedcookbook,installing/Installingacookbookapplying,torole/Applyingrecipestorolesexecuting/Startingoutsmalltesting/Testingrecipestesting,withChefSpec/Testingrecipesbuilding/Buildingyourrecipetests,executing/Executingtestsfailures/Understandingfailurescreating,inChefshell/Creatingarecipeintheshell

recipesabout/Recipes,Recipesstartingstates/Recipesdeveloping/Developingrecipeswriting/Writingrecipessimpleservice,installing/Installingasimpleservicecomplicatedrecipe,actions/Gettingmoreadvancedenhancing,withsearchmethod/Searchinginsiderecipesdatabags,searchingfrom/Searchingyourdatabagsfromarecipe

recipes,executingwithChefshellabout/ExecutingrecipeswithChefshell,Interactivelyexecutingrecipesrecipemode/Creatingarecipeintheshellattributemode/Creatingarecipeintheshellnodeattributes,defining/Definingnodeattributesattributes,setting/Settingattributesattributes,accessing/Accessingattributesconfigurationblocks,using/Usingconfigurationblocks

RedHatEnterpriseLinuxChefserver,installingon/InstallingonRedHatEnterpriseLinux

resource/GettingtoknowChefbuilding/Buildingaresourcepackageresource/Buildingaresourcedefining/Definingtheresource

provider,implementing/Implementingtheprovidermodifying/Modifyingresourcesexistingresource,loading/Loadinganexistingresourceupdatedstatus,declaring/Declaringthataresourcewasupdated

resourcesabout/Resourcesbuilt-inresources/Resourcesusing/Usingresourcesdefaultbehavior,overriding/Overridingadefaultbehavior

reusableresources,defininginChefbenefits/Definingafullapplicationtemplate

roleabout/GettingtoknowChef,Rolesdefining/Definingrolesimplementing/Implementingaroleapplying,torecipes/Applyingrecipestorolesmapping,tonode/Mappingyourrolestonodes

role,definingabout/Definingroleswebapplicationservicerole/Awebapplicationserviceroleimage-processingrole/Animage-processingroleimagesearchrole/AnimagesearchrolePostgreSQLservicerole/APostgreSQLserviceroleSolrservicerole/ASolrserviceroleOpenSSHservicerole/AnOpenSSHservicerole

rolesdefining/Definingrolesbaseserverrole,creating/Creatingthebaseserverroledatabaseserverrole,creating/Creatingthedatabaseserverrolewebserverrole,creating/Creatingthewebserverrole

roles,usingwithChef-soloabout/UsingrolesanddatabagswithChef-solocustomJSONdata,injecting/InjectingcustomJSONdatacustomnodename,providing/Providingacustomnodename

RSpecabout/RSpecandChefSpec/RSpecandChefSpectestinglevels/RSpecandChefSpeccomparing,withtestinglibraries/ComparingRSpecwithothertestinglibrariesfailures/Understandingfailures

Rubydependencies,locking/LockingyourdependenciesinRuby

Rubygemmechanism/TheRubygemRubyVersionManager(RVM)

about/TheRubygemURL/Managinggems

runlist/GettingtoknowChefrunner/Testingformultipleplatforms

Ssearchmethod

used,forenhancingrecipes/Searchinginsiderecipessearchqueries

used,formanagingmultiplemachines/Managingmultiplemachineswithsearchqueries

secretkeys,databagsmanaging,withsearchqueries/Secretkeys

serverlistsubcommand/Terminatinganinstanceservice-orientedarchitecture(SOA)/VagrantandChef-solosetup

describing/Describingthesetupshelluserrecipe

evolution/Evolutionofashelluserrecipesoftware

deploying,withChef/DeployingsoftwarewithChefdeploying/Deployingyoursoftware

softwaredeploymentupdates,deployingmanually/Manuallydeployingupdatesautomating/Automatingdeployment

solomode,Chefshell/ThesolomodeSolrservicerole

defining/ASolrservicerolesourcecode

checking/CheckingthesourcecodeSSHkeys

distributing/DistributingSSHkeysauthorizedkeys,templating/Templatingtheauthorizedkeysdeploymentkeys,adding/Addingdeploymentkeys

standalonemode,Chefshell/Thestandalonemodesupervisordservice

URL/Keepingyourapplicationrunning

Ttemplates

about/Templatesneedfor/Whyusetemplates?ERBprimer/AquickERBprimertemplateresource/Thetemplateresourcevariables/Thetemplatevariablesvariables,passingto/Passingvariablestoatemplatecomputedconfiguration,accessing/Accessingcomputedconfigurationssearchingfor/Searchingfortemplatessearchorder/Searchingfortemplates

terminology,Chefnode/TerminologyChefservice/Terminologyworkstation/Terminologyrecipe/Terminologycookbook/Terminology

test-driven-development(TDD)/TestingbasicsTestKitchen

using/UsingTestKitcheninstalling/InstallingTestKitchenused,fortesting/TestingwithTestKitchentasks/TestingwithTestKitchencookbook,building/Buildingasimplecookbookcookbook,preparingfor/Preparingyourcookbookforthekitchencookbook,testing/Testingyournewcookbook

UUbuntu

Chefserver,installingon/InstallingonUbuntuupdated_by_last_actionmethod/Declaringthataresourcewasupdatedusercookbook

enhancing/Enhancingyourusercookbookusermanagement

Chef/Managingusersshelluserrecipe,evolution/Evolutionofashelluserrecipedata,storingindatabags/Storingdataindatabagsrecipe,enhancingwithsearchmethod/Searchinginsiderecipesusercookbook,enhancing/EnhancingyourusercookbookSSHkeys,distributing/DistributingSSHkeys

usersadding/Addingusersmanaging/Managingusers

VVagrant

about/VagrantandChef-soloinstalling/InstallingVagrantURL/InstallingVagrantused,forprovisioningnewhost/ProvisioninganewhostwithVagrantimage,booting/BootingyourVagrantimagecombining,withChef-solo/CombiningVagrantwithChef-solo,UnderstandingthelimitationsofChef-solo,TellingChef-solowhattorun

VagrantCloudURL/ProvisioninganewhostwithVagrant

Vagrantimagebooting/BootingyourVagrantimage

Wwebapplicationservicerole

defining/Awebapplicationservicerolewebserver

configuring/Configuringthewebserverwebserverrole

creating/Creatingthewebserverrolewhy-runmechanism/Implementingtheproviderworkstation/Terminology,GettingtoknowChef

YYAML/DeployingsoftwarewithChef