the-eye.eu...table of contents chef essentials credits about the author about the reviewers support...

ChefEssentials

TableofContents

ChefEssentials

Credits

AbouttheAuthor

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffersandmore

WhySubscribe?

FreeAccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Errata

Piracy

Questions

1.InstallingChef

Terminology

WorkingwithChef

Installingchef-solo

TheRubygem

Managinggems

Verifyingthatchef-soloworks

InstallingaChefserver

Requirementsandrecentchanges

Installationrequirements

Whatyouwillbeinstalling

Gettingtheinstaller

Installationoutline

InstallingonUbuntu

Downloadingthepackage

Installingthepackage

InstallingonRedHatEnterpriseLinux

Downloadingthepackage

ConfiguringaChefserver

Understandinghowchef-server-ctlworks

What’shappeningonmyserver?

Verifyingthattheservicesarerunning

Validatingthatyourserviceisworking

Ensuringthatyourknifeconfigurationworks

Summary

2.ModelingYourInfrastructure

GettingtoknowChef

Modelingyourinfrastructure

Definingroles

Awebapplicationservicerole

Animage-processingrole

Animagesearchrole

APostgreSQLservicerole

ASolrservicerole

AnOpenSSHservicerole

Implementingarole

Determiningwhichrecipesyouneed

Installingacookbook

Applyingrecipestoroles

Mappingyourrolestonodes

Converginganode

Environments

Organizingyourconfigurationdata

Exampleattributedata

Databags

Knowingwhentousedatabags

Large-scaleinfrastructure

Summary

3.IntegratingwiththeCloud

Leveragingthecloud

AmazonEC2

InstallingtheEC2knifeplugin

SettingupEC2authentication

Provisioninganinstance

Bootstrappingtheinstance

Terminatingtheinstance

RemovingtheChefnode

RackspaceCloud

Provisioninganinstance

Terminatinganinstance

RemovingtheChefnode

Summary

4.WorkingwithCookbooks

Attributes

Multipleattributefiles

Supportingmultipleplatforms

Loadingexternalattributes

Usingattributes

Metadata

Recipes

Resources

Usingresources

Overridingadefaultbehavior

Templates

Whyusetemplates?

AquickERBprimer

ExecutingRuby

Variablereplacement

Thetemplateresource

Thetemplatevariables

Passingvariablestoatemplate

Accessingcomputedconfigurations

Searchingfortemplates

Definitions

Recipes

Developingrecipes

Writingrecipes

Startingoutsmall

Installingasimpleservice

Gettingmoreadvanced

Summary

5.TestingYourRecipes

Testingrecipes

RSpecandChefSpec

Testingbasics

ComparingRSpecwithothertestinglibraries

UsingChefSpec

GettingstartedwithChefSpec

InstallingChefSpec

LockingyourdependenciesinRuby

CreatingasimplerecipeandamatchingChefSpectest

WritingaChefSpectest

Buildingyourrecipe

Executingtests

Understandingfailures

Expandingyourtests

Multipleexamplesinaspectest

Testingformultipleplatforms

Summary

6.FromDevelopmenttoDeployment

Describingthesetup

DeployingsoftwarewithChef

Configuringyourlocalenvironment

ModelingasimplePythonapplication

Managingthecookbooks

Downloadingcookbooks

Lookingatthedatabaserecipe

Lookingatyourapplicationdeploymentcookbook

Preparingthedirectories

ConstructingyourPythonvirtualenvironment

Checkingthesourcecode

Installinganyextradependencies

ManagingdependenciesinChef

Managingdependencieselsewhere

UsingPython’srequirementsfile

Configuringyourapplication

Keepingyourapplicationrunning

Definingroles

Creatingthebaseserverrole

Creatingthedatabaseserverrole

Creatingthewebserverrole

Addingusers

ProvisioningEC2instances

Configuringthedatabasehost

Configuringthewebserver

Deployingyoursoftware

Manuallydeployingupdates

Automatingdeployment

Summary

7.BeyondBasicRecipesandCookbooks

Managingusers

Evolutionofashelluserrecipe

Storingdataindatabags

Creatingadatabagforusers

Searchingfordata

Searchinginsiderecipes

Enhancingyourusercookbook

DistributingSSHkeys

Templatingtheauthorizedkeys

Addingdeploymentkeys

Writingcustomextensions

Developingacustomdefinition

Organizingyourcode

WritingadefinitionforusingPIP

Definingafullapplicationtemplate

Buildingaresource

Definingtheresource

Implementingtheprovider

Modifyingresources

Loadinganexistingresource

Declaringthataresourcewasupdated

Workingwithdatabags

Securingyourdatabags

Secretkeys

Encryptingyourdata

Decryptingyourdata

Storingkeysonnodes

Searchingyourdata

Searchingyourdatabagswithknife

Searchingyourdatabagsfromarecipe

Queryingyourdata

Managingmultiplemachineswithsearchqueries

Summary

8.ExtrasYouNeedtoKnow

VagrantandChef-solo

InstallingVagrant

ProvisioninganewhostwithVagrant

BootingyourVagrantimage

CombiningVagrantwithChef-solo

UnderstandingthelimitationsofChef-solo

ConfiguringChef-solo

TellingChef-solowhattorun

UsingrolesanddatabagswithChef-solo

InjectingcustomJSONdata

Providingacustomnodename

GettingtoknowtheChefshell

UsingtheChefshell

Thestandalonemode

Thesolomode

Theclientmode

InteractingwiththeChefserverusingtheshell

Interactingwithdata

Searchingyourdata

Editingyourdata

Transformingdata

ExecutingrecipeswithChefshell

Creatingarecipeintheshell

Definingnodeattributes

Settingattributes

Accessingattributes

Usingconfigurationblocks

Interactivelyexecutingrecipes

DebuggingwiththeChefshell

Usingthebreakpointresource

Integrationtesting

UsingTestKitchen

InstallingTestKitchen

TestingwithTestKitchen

Buildingasimplecookbook

Preparingyourcookbookforthekitchen

Testingyournewcookbook

Provisioningtheinstance

Convergingthenewlycreatedinstance

Writingasimpletest

Combiningallthesteps

ExtendingChef

WritinganOhaiplugin

AnoteaboutwritingOhaiplugins

ChefwithCapistrano

Automationandintegration

Automatedupdatesanddeployments

Summary

ChefEssentials

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:September2014

Productionreference:1190914

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78398-304-9

www.packtpub.com

CoverimagebyPrashantTimappaShetty(<sparkling.spectrum.123@gmail.com>)

CreditsAuthor

JohnEwart

Reviewers

JoshuaBlack

LaurenMalhoit

EricMaxey

CommissioningEditor

EdwardGordon

AcquisitionEditor

LlewellynRozario

ContentDevelopmentEditor

GovindanK

TechnicalEditor

ShubhangiDhamgaye

CopyEditors

RoshniBanerjee

MradulaHegde

GladsonMonteiro

ProjectCoordinator

SageerParkar

Proofreaders

SimranBhogal

MariaGould

AmeeshaGreen

LindaMorris

Indexer

RekhaNair

ProductionCoordinator

ArvindkumarGupta

CoverWork

ArvindkumarGupta

AbouttheAuthorJohnEwartisasystemarchitect,softwaredeveloper,andlecturer.Hehasdesignedandtaughtcoursesatavarietyofinstitutions,includingtheUniversityofCalifornia,TheCaliforniaStateUniversity,andlocalcommunitycolleges.Thesecoursescoverawiderangeofcomputersciencetopics,includingJava,datastructuresandalgorithms,operatingsystemsfundamentals,UnixandLinuxsystemadministration,andwebapplicationdevelopment.Inadditiontoworkingandteaching,hemaintainsandcontributestoanumberofopensourceprojects.HecurrentlyresidesinRedmond,Washington,withhiswife,Mary,andtheirtwochildren.

AbouttheReviewersJoshuaBlackhasbeenworkingwithcomputersprofessionallyfor20years.Hehasawiderangeofexperienceandexpertise,whichincludessystemsandnetworkadministration,mobileappdevelopment,andproductionwebapplications.HeearnedaBSdegreeinComputerSciencewithaminorinMathfromCaliforniaStateUniversity,Chico,in2005.HecurrentlyresidesinChico,California,withhiswife,Rachel,andtheirfourchildren.

LaurenMalhoithasbeeninthefieldofITforover10yearsandhasacquiredseveraldatacentercertifications.She’scurrentlyatechnicalvirtualizationarchitect,specializinginvirtualizationandstorageindatacenter.ShehasbeenwritingforafewyearsforTechRepublic,TechRepublicPro,andVirtualizationSoftware.AsaCiscoChampion,EMCElect,VMwarevExpert,andPernixPro,shestaysinvolvedinthecommunity.Shealsohostsabi-weeklytechnologypodcastcalledAdaptingIT(http://www.adaptingit.com/).ShehasbeenadelegateforTechFieldDayseveraltimesaswell.Sherecentlypublishedherfirstbook,VMwarevCenterOperationsManagerEssentials,PacktPublishing.

EricMaxeyhasavariedbackgroundinwritingsoftware,includingmakingconsolevideogamesandanalyzingtheadrevenuedata.Heranasmallbusinessthatpioneeredanewkindofbitcoinminingpool,whichhasnowbecomesomethingofastandard.Whennotjackedintothemetaverse,helikestoworkonelectricbicyclesandridethemoffroad.

www.PacktPub.com

Supportfiles,eBooks,discountoffersandmoreYoumightwanttovisitwww.PacktPub.comforsupportfilesanddownloadsrelatedtoyourbook.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

http://PacktLib.PacktPub.com

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcanaccess,readandsearchacrossPackt’sentirelibraryofbooks.

WhySubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,printandbookmarkcontentOndemandandaccessibleviawebbrowser

FreeAccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandviewnineentirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

PrefaceChefisanindispensabletooltomanageyourinfrastructure.Itconsistsofasetoftoolsthataredesignedtoworktogethertoenableyoutomodelandmanageyoursystems.Thisisalargespacetofill,andChefprovidesyouwiththetoolstodothisinaveryflexibleandpowerfulway.Itachievesthisthroughacombinationofservices,endhostagents,awebinterface,andcommand-linetoolsthatworkinunisontodeliveranincrediblesuiteoftools.

Chef’sservicesareresponsibleforstoring,managing,anddistributingdataaboutyourinfrastructurethroughanAPI.Endhostsoftwareagentsthatrunonnodes(managedsystems)areresponsibleforperformingupdatestosystems,andthewebinterface,alongwithcommand-linetools,allowsanadministratortoeditandconsumeinformationthatisvendedbytheAPIservice.

OneofthemostattractivefeaturesofChefisthatyoucanleverageitsAPItoeasilyintegrateexistingtools,oryoucandevelopnewtoolstomeetspecificneeds.AnyorganizationwithamoderatenumberofdeveloperresourcescanharnessthepowerofCheftomanagetheirsystems.Forexample,onecaneasilybuildsoftwaretoimportdatafromChefintoareportingtoolofsomeformanddynamicallyreconfigureinfrastructurebasedonathird-partytool’soutput—thesky’sthelimit.ThisisincrediblyvaluabletoanybodywhohasanexistinginfrastructurebecauseitprovidesaconvenientpathtointegrateChefintotheirenvironment.

ThereareanumberofwaystoaccessChef.Thequickestwayforasingleusertomanagehis/herinfrastructure(virtualmachines,ahandfulofhosts,andsoon)istouseChef-solo,aproductgearedtowardssingle-userenvironments.Inasmallenvironment,settingupahostedserverisagoodwaytomanageinfrastructureautomationamongteammembers.Ifyouneedto,youcanconfigurethehostedenvironmentasahighlyavailablesystemusingloadbalancersandothertechnologies.Alternatively,ifhostingtheserviceyourselfisnotanoption,youcanusehostedChef,asoftware-as-a-service(SaaS)model,thuspayingforaccesstoahostedservice.

Configurationmanagementsoftwarewascreatedtofillaneed—managinginfrastructureisachallengingtask.Regardlessofthescaleyouoperateon,keepingtrackofsoftwareversions,upgradingsystems,andgeneratingconsistentconfigurationdataisalotofwork.Itistemptingtoupdateaconfigurationfileononesystem,onlytoforgettocommitthosechangessomewhere,ortoapplythemtoexistingorfuturehosts.Thisisveryconvenient,butitquicklyleadstoinconsistencybetweenhosts.Whenyouareworkingwithonlyoneortwohosts,thismaybeacceptable.Assuch,asystemgrowsfromafewserverstodozens,hundreds,orpossiblythousands—thistypeofsystemmanagementdoesnotscaleduetotimerequirementsandconfigurationerrorsthatresultfromsizeandcomplexity.

Considerascenariowhereyouaremigratingadatabaseservertoanewhost.Thiswouldinvolve:bringingupanewhost,installingalloftherequiredsoftwareonyourlisttoensureithasparitywiththeoldserver(youdidkeepalist,right?),ensuringthatyourdatabaseserverwasconfiguredwiththesameoptions,puttingthecorrectfirewallrulesin

place,tuningthefilesystem,settingupmonitoringtools,updatingDNSrecordsorchangingwebapplicationconfigurationstopointtothenewhost,andsoon.Nowimaginethat,insteadofoneserverinonedatacenter,youhave10databaseserversin10datacenters,eachwiththeirownIPranges,hardwareconfigurations,andnetworkingrules.Situationssuchasthisareexactlywhysystemconfigurationmanagementsoftwarepackagesweredeveloped:tomakethelivesofsystemadministratorsandengineersmucheasier.

Thisscenario,andmanyotherslikeit,iswhereChefisindispensable.Havingtheabilitytodescribeyourhosts,configurationdata,androles,andthenapplythatacrossasmanyhostsasyoulikemeansthatyoucanmanagelargefleetsofhostsjustaseasilyasyoucanmanageoneortwo.

WhatthisbookcoversChapter1,InstallingChef,introducesyoutothearchitectureofChef,variousinstallationmethods,andaguidetosettingupChef(soloandself-hosted).ItincludesinformationonusinghostedChef(andwhatthatmeansforyourteam)andVagrantwithChef-solo.

Chapter2,ModelingYourInfrastructure,introduceshowtomodelyourinfrastructurewithChefusingyournewlyinstalledsystem.Thischapterwillcovermodelingenvironments,smallandlarge,aswellashowtointegratewithcloudtechnologiesusingChef(AWS,RackspaceCloud,andsoon).

Chapter3,IntegratingwiththeCloud,covershowChefhelpsyouscaleyourinfrastructureusinganycombinationofphysical,virtual,andcloud-hostedsystems.ThischapterdiscusseshowtouseCheftoprovisionandmanagehostsusingcloudprovidersaseasilyasyourlocalsystemsincludingAWSandRackspaceCloud.

Chapter4,WorkingwithCookbooks,covershoweveryChefneedscookbooks—onceyoursystemsarepartofyourChef-managedfleet,youcanbegincollecting,developing,andapplyingrecipestoyourhosts.Itincludesin-depthexplanationsofthestructureanddevelopmentofcookbooksandrecipes,aswellashowtotest,publish,andsharethem.

Chapter5,TestingYourRecipes,focusesononecompellingreasontouseCheftoconfigureyourinfrastructure,thatis,recipesarewritteninRubycodeandcanbetestedasanyprogramwouldbetested.Here,youwilllearnhowtotestyourrecipesthroughavarietyoftestingmechanisms.

Chapter6,FromDevelopmenttoDeployment,covershowtotakeacustomapplicationfromdevelopmenttoaproductiondeploymentwithChef.Itcontainsacompleteexamplethatincludesprovisioningawebserver,databaseserver,andusersaswellasdeployingcodefromsourcecontrol.

Chapter7,BeyondBasicRecipesandCookbooks,delvesintodevelopingextensionstoChefthroughadvancedconcepts,includingcustomprovidersandresourcetypes,usingtheChefsearchengine,advancedscripting,andmore.

Chapter8,ExtrasYouNeedtoKnow,expandsyourknowledgeofhowtoleverageChefforinfrastructureautomation,complexsystemsintegration,andsecurelystoringanddistributingsensitivedatawithChef.

WhatyouneedforthisbookThisbookassumesthatyouarefamiliarwithatleastoneprogramminglanguage(itdoesnotneedtobeacompiledlanguage,andknowledgeofaninterpretedlanguagewillbesuitable.ChefusesRubyforitsdynamic,scriptablecomponentsandanyexperiencewithRubywillbevaluable.However,havingastrongunderstandingofprogramlogicwillprovideyouwiththebackgroundtobeproductivewithChef.

ForthosewhoarenotexpertswithRuby,therewillbeawidearrayofexamplelistingsthatcanbecopieddirectlyandexecutedaspartofthebook’sofferings.ThiswillenableyoutousetheexampleswithoutanypreviousRubyexperience.However,aworkingknowledgeofRubywillbeneededinordertoexpandonthebook’scodeexamplesorwhilewritingyourownrecipesfromscratch.

YouwillbewalkedthroughthestepsrequiredtoinstallChefonaLinux-basedhost.Inordertobeimmediatelysuccessful,youwillneedadministrativeaccesstoahostthatrunsamodernversionofLinux;Ubuntu13.10iswhatwillbeusedfordemonstrationpurposes.Ifyouareamoreexperiencedreader,thenarecentreleaseofalmostanydistributionwillworkjustaswell(butyoumayberequiredtodoalittlebitofextraworkthatisnotoutlinedinthebook).IfyoudonothaveaccesstoadedicatedLinuxhost,avirtualhost(orhosts),runninginsideofvirtualizationsoftware,suchasVirtualBoxwillwork.

Additionally,youwillneedaccesstotheInternettodownloadsoftwarepackagesthatyoudonotalreadyhave,aswellasaninstallationoftheRubyprogramminglanguageVersion1.9orhigher.

WhothisbookisforThisbooktargetsdevelopersandsystemadministratorswhoneedtomanageinfrastructureandarelookingtoautomatetheirsystemmanagement.Thisincludesinfrastructureranginginsizefromsmall-scaleinstallationswithahandfulofhoststomulticontinentcorporateITsystemswithhundredsoreventhousandsofhosts.Anybodywhosejobinvolvesmaintainingsystemswillbenefitfromtheconceptsbeingcovered.

ConventionsInthisbook,youwillfindanumberofstylesoftextthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestyles,andanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Thesearchmethodhasasimilarformattotheknifecommand.”

Ablockofcodeissetasfollows:

all_users=search(:users,'id:*')

users_s=search(:users,'id:s*')

all_nodes=search(:node,'*')

Anycommand-lineinputoroutputiswrittenasfollows:

$knifedatabagshowcredentialsaws

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,inmenusordialogboxesforexample,appearinthetextlikethis:“Oncethere,atablabeledChefServerwillbepresentatthetopofthepage.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedormayhavedisliked.Readerfeedbackisimportantforustodeveloptitlesthatyoureallygetthemostoutof.

Tosendusgeneralfeedback,simplysendane-mailto<feedback@packtpub.com>,andmentionthebooktitleviathesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideonwww.packtpub.com/authors.

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyouwouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheerratasubmissionformlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedonourwebsite,oraddedtoanylistofexistingerrata,undertheErratasectionofthattitle.Anyexistingerratacanbeviewedbyselectingyourtitlefromhttp://www.packtpub.com/support.

PiracyPiracyofcopyrightmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.Ifyoucomeacrossanyillegalcopiesofourworks,inanyform,ontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthors,andourabilitytobringyouvaluablecontent.

QuestionsYoucancontactusat<questions@packtpub.com>ifyouarehavingaproblemwithanyaspectofthebook,andwewilldoourbesttoaddressit.

Chapter1.InstallingChefBeforeyoucanstartusingChef,youwillneedtoinstallit.HereyouwillfindaguidetoinstallChef,andbecauseChefrequiresRuby,someRubyconceptsaswell.Thischapterdiscussesthefollowing:

KeyterminologyandconceptsrelatedtoChefAnoverviewofChef’sarchitectureWorkingwithRubygemsInstallingchef-solo(alocal-onlyenginetouseChef)Abriefexampleonusingchef-soloInstallingtheChefserveronyourownhostVerifyingyourChefinstallation

TerminologyAswithanyothertechnology,Chefhasitsownterminology.Asyouwillsee,Chef’snomenclatureisamixoftechnologicalterms(nodes,workstations,servers,roles,andsoon)andcookingterms(cookbooks,recipes,andsoon).Therearethreeprimaryactorsthatweareconcernedwithatthispoint:nodes,theChefservice,andworkstations.

Node:Anodeisaclientthatappliesrolesandrecipes,asdescribedbytheadministratorintheChefservice(thatis,aserverinyourenvironmentthatisbeingconfiguredviaChef).Thesearetheconsumersoftheconfiguration,theelementsofyourinfrastructure.TheycanbephysicalorvirtualmachinesandcanrunonLinux,Windows,ortechnicallyanyothersystemthatiscapableofrunningRuby(somesystemsmaynotbesupportedoutoftheboxbyChef).Chefservice:TheChefserviceisamulticomponentsystemthatcombinesseveralservicestoprovideitsfunctionality.TheprimaryfunctionalcomponentsareanAPIservice,full-textsearchingviaSolr,persistentstorageusingPostgreSQL,andRabbitMQforinterservicecommunication.Additionally,thereisawebinterfacethatprovidesagraphicaltooltomanagesystemdata.Clients(nodes)usetheAPIservicetodeterminewhichrolesandrecipestoapply,andknife(acommand-linetool)usestheAPItoallowanadministratortoeditandmanagetheirChefconfiguration.Workstation:Aworkstationisahostthatisusedtoissuecommands.AworkstationcanbeaseparatehostoutsideofyourChefserviceinstallation,amanagednode,ortheserverthattheChefcomponentsarerunningon.Thereareavarietyofcommand-linetoolsthatareprovidedtointeractwiththeservice,whichwillbeinstalledontoyourworkstation(s).Recipe:Arecipeisascriptthatdescribesasetofstepstotaketoachieveaspecificgoal.Asanexample,arecipemightdescribehowtodeployyourcustomsoftware,provisionadatabase,oraddahosttoaloadbalancer.Cookbook:Acookbookisacollectionofrecipesthatareusedtocollectivelydescribehowtoinstall,configure,andmanagevariousaspectsofyourinfrastructure.Forexample,acookbookmightdescribehowtoprovisionMySQL,PostgreSQLorApache,manageusers,installprinters,orperformanyothersystemtasks.

WorkingwithChefForsingleusersetups,chef-soloisaversionofthechef-clientthatallowsyoutouseChefwithoutrequiringaccesstoaserver.Chef-solorunslocallyandrequiresthatacookbookandanyofitsdependenciesbepresentonthenodebeingmanaged.Asaresult,chef-soloprovidesalimitedsubsetofthefullchefservermodeofoperation.Mostofthefeaturesthatchef-soloismissingrevolvearoundsearchandcentralizeddatamanagement,whicharenotcriticalformanagingvirtualmachinesorasmallcollectionofnodes.Theinstallationandmaintenanceissimple,butthefeaturesetissmaller.

InstallingtheChefserverwillgiveyouaccesstothefullsetofCheffunctionality.ThismoderequiresaccesstoaLinux-basedhostthatisnetwork-accessiblebythenodesandworkstationsthatwillinteractwiththesystem.ThankstotherecenteffortfromthefolksatChef(formerlyOpscode),theprocessofinstallingChefhasbeengreatlysimplified.Thebenefitsofthisinstallationmodelarethatyougetcentralizedmanagement,search,userauthentication,andsuch,butatthecostofmanagingyourownservice.

IfyouneedthefeaturesofChefbutdonotwanttomaintainyourownserver,hostedChefisagreatoptionforyou.HostedChef(https://manage.opscode.com/signup)givesyouallthefeaturesofaself-hostedChefbutwithouthavingtoworryaboutupgrades,extrahardware,orsystemavailability.Forasmallinfrastructure(uptofivehosts),hostedChefisfreeandagreatwaytogetstarted.Beyondthis,planshaveamonthlyfee,andthepricewillvaryaccordingtothenumberofhostsyouwanttomanage.

Installingchef-soloChef-soloisdesignedforindividualswhodonotneedahostedinstallationforalarge-scaleinfrastructuremanagement.Typicalusecasesofchef-soloincludedevelopersmanagingvirtualmachines,testinstallations,orsmall-scaleinfrastructuremanagement.Theinstallationofchef-soloisassimpleasinstallingasingleRubygem.

TheRubygemForthosewhoarenotintimatelyfamiliarwithRuby,aRubygemisamechanismtopackage,deliver,andmanageRubycode.Thesepackagesmaybelibrariesthatprovidefunctionalityfordevelopers,ortheymaybecomposedonlyofscriptsandtools.Chef-solois,likemanythingsinlife,somewhereinthemiddle.Thegemcontainsasetoflibrariesthatmakeupthecorefunctionalityaswellasasuiteofscriptsthatareusedbyendusers.BeforeyouinstallChef,youshouldconsiderinstallingRubyVersionManager(RVM),rbenv,chruby,oranotherRubymanagerofyourchoicetokeepyourgemcollectionsisolated.

ManaginggemsAgreattooltomanageyourgemsisRVM.ThesimplestwaytoinstallRVMistousetheinstallationscriptprovidedbythedevelopmentteamontheRVMwebsite(http://rvm.io).Thefollowingcommandwilldownloadthescriptandpipeitthroughbash:

curl-sSLhttps://get.rvm.io|bash-sstable

Onceitisinstalled,youwillinitiallyneedtoincludeRVM’sfunctionalityinyourshell:

source~/.rvm/scripts/rvm

Additionally,youmightneedtoaddthepreviouscommandlinetoyourshell’sstartupscripts(suchas~/.bashrcor~/.zshrc,dependingonwhichshellyouuse).OnceRVMisinstalled,youwillwanttoinstallarecentversionofRuby,forexample,Ruby1.9.3:

rvminstall1.9.3

OnceRuby1.9isinstalled,youwillwanttocreateagemset.AgemsetisRVM’swayofisolatinggemsinsideacontainer,anditwillprovideyouwithaplacetoinstallgemsinsuchawaythattheywillnotconflictwithothergems.Thishasthebenefitofallowingyoutoinstallanythingyouwant,withoutrequiringadministrativeprivilegesandkeepinggemsfromconflictingwitheachother.Agemsetcanbecreatedusingthefollowingcommand:

rvmuse1.9.3@chef--create

Thepreviouscommandwillsimultaneouslycreatethegemsetnamedchef(ifitdoesnotexist)foryourinstallationofRuby1.9.3andthensetitastheactivegemset.Onceyoustartusingthisnewgemset,youwillwanttoinstalltheChefgem—thiscontainschef-soloandallthecommand-linetoolsyouneedtoworkwithChef—usingthegemcommand-linetool:

geminstallchef

Verifyingthatchef-soloworksNowthattheChefgemisinstalled,itistimetoverifythateverythingisworkingfine.Inordertousechef-solo,youneedtogivethefollowinginformationtoit:

Whatrecipestoapplybyprovidingarunlistinafilenamednode.jsonWhattherecipesare—thesearestoredincookbooksthatarefoundinthecookbooksdirectoryHowtofindthecookbooksandtherunlistviaafilenamedsolo.rb

Forsimplicity,wewillstoreallofthesefilesinsideofthechefdirectoryinyourhomedirectory.YouarefreetoputthingswhereyouseefitasyoubecomemorecomfortableworkingwithChef.

Inordertoexerciseournewtool,wewilldosomethingsimple:we’llwritearecipethatwillcreateanexample.txtfileinyourhomedirectory.Therecipewecreatewillbecalledcreate_file,andwe’llputthatrecipeinsideacookbook,whichwillbenameddemo.

First,createthedirectorythatwillcontainthedemocookbook’srecipes(andanyinbetween):

user@host:~$mkdir-p~/chef/cookbooks/demo/recipes

Next,addthefollowingcodetoafile,create_file.rb,locatedinthedemocookbookdirectoryyoucreatedat~/chef/cookbooks/demo/recipes:

file"#{ENV['HOME']}/example.txt"do

action:create

content"Greetings#{ENV['USER']}!"

ThistellsChefthatwewanttocreateafile,$HOME/example.txt.ItscontentsshouldbeGreetings$USER,where$USERwillbereplacedwiththevalueof$USER,typicallytheloginnameofwhoeverisexecutingtherecipe.

TipForthoseunfamiliar,UNIX(andWindowsaswell)usesenvironmentvariablesasamechanismtoexchangedatabetweenprocesses.SomeenvironmentvariablesaresetwhentheuserlogsintothesystemsuchasHOME,USER,andavarietyofothers.ThesevariablesareavailableinRubyusingtheENVhash,wherethekeysarethevariablenames.InaUNIXshell,theseareaccessedusingthe$prefix.So,theuser’shomeisreferredtoas$HOMEintheshellandENV['HOME']insideRuby.

NowwewillneedtocreateaJSONdocumentthatdescribeswhatchef-soloshouldexecute.JSONisanacronymforJavaScriptObjectNotation,andChefusesJSONextensivelybecauseitiseasytoparse,humanreadable,andeasytogeneratefromallsortsoftoolsandlanguages.Createafile,node.json,locatedinourworkdirectory(~/chef/inthiscase)andaddthefollowingcontentinordertotellChefthatwewanttoexecutethenewlycreatedcreate_filerecipeinthedemocookbook:

"run_list":[

"recipe[demo::create_file]"

Here,wearedefiningthenodeashavingarunlist,whichisjustanarrayofthingstodo,andthattherunlistcontainsonerecipe,create_file,whichitcanfindinthedemocookbook(thegeneralformofarecipebeingcookbook::recipe).

Finally,we’lltellChefwheretofindthefileswejustcreatedusingasolo.rbfilethatwewillstoreinourworkingdirectory(~/chefinourcase):

CHEF_ROOT="#{ENV['HOME']}/chef"

file_cache_path"#{CHEF_ROOT}"

cookbook_path"#{CHEF_ROOT}/cookbooks"

json_attribs"#{CHEF_ROOT}/node.json"

Nowthatyouhavepopulatedtherequiredconfigurationfiles,youcanrunchef-soloandexecutetherunlistspecified.Inourcase,therunlistisdefinedasonlyonerecipe,create_file,butcanbeassimpleorascomplexasneeded.ThepreviousconfigurationtellsCheftoloadthenodeconfigurationfromthefilenode.jsontolookforcookbooksin~/chef/cookbooks/andtostoreanystatedatain~/chef/.Inordertoexecutethesecommands,youwillwanttorunchef-solo:

chef-solo-c~/chef/solo.rb

The-coptiontellschef-solowhichscriptcontainstheconfiguration.Onceyoudothis,youwillseetheactionsthatyourrecipeisperforming:

StartingChefClient,version11.8.2

CompilingCookbooks…

Converging1resources

Recipe:demo::create_file

*file[/home/user/example.txt]actioncreate

-createnewfile/home/user/example.txt

-updatecontentinfile/home/user/example.txtfromnonetob4a3cc

---/home/user/example.txt 2014-01-2023:59:54.692819000-0500

+++/tmp/.example.txt20140122-13411-1vxtg7v 2014-01-20

23:59:54.692819000-0500

@@-1+1,2@@

+Greetingsuser!

ChefClientfinished,1resourcesupdated

Onceitiscompleted,youwillseethat~/example.txtcontainsthegreetingthatyoudefinedintherecipe.Nowthatyouhavesuccessfullyusedchef-solo,let’smoveontotheChefservice.

InstallingaChefserverIfyourteamneedstohavecentralizedinfrastructuremanagementanddoesnotwanttouseahostedplatform,thenaself-installedChefserverisaperfectfit.ThisinstallationguideassumesthatyouwillberunningtheChefserveronasupportedLinux-basedsystem.

TheChefservicecomponentscanbeinstalledonasinglemachinewithoutanyissue.Installingitonasinglehostwilllimityourabilitytoscaleorbehighlyavailable,butwillprovideaverysimplepathtogettingstartedwiththeChefservice.

RequirementsandrecentchangesSincetheChefserviceisdesignedtobeamultiuserplatformandprovidesfunctionalitiesthatchef-solodoesnotoffer,theinstallationismorecomplexandinvolvesmoresoftwaretoachievethisfunctionality.ServicessuchasSolrforfull-textindexingandPostgreSQLfordatastoragecanbeasignificantresourceforconsumers,soyouwillwanttoinstallChefonahostwithsufficientmemoryanddiskspace.Asystemwith2GBofmemoryand5-10GBofdiskspaceavailablewillbeplentyforasmalltomediumsizedinstallation.Youwillneedmoreresourcesasyourrequirementsfordatastorageandindexingincreaseovertime,soplanaccordingly.

Additionally,forthosewhohaveinstalledtheChefserverbefore,theinstallationpathhasbeengreatlysimplified.InadditiontoreplacingCouchDBwithPostgreSQLastheprimarydatastorageengine,thereisnowasingleomnibusinstallationpackageforChefthatinstallsalloftherequirementsforChefatasinglelocationsothatitoperatesinisolationanddoesnotrequiredependenciestobeinstalledseparately.

InstallationrequirementsInordertoinstalltheChefservice,youwillneedtohavethefollowing:

AsystemrunningasupportedLinuxvariant(64bitUbuntuLinux10.04through12.10or64bitRedHatEnterpriseLinux5or6)—thiscanbephysicalorvirtual.Ifyoudonothavethelocalresourcesforthis,AWSorRackSpacecloudserversaregoodoptions.Anetworkconnectiontothehostinordertodownloadtheinstaller.Administrativeprivileges(usingsudoordirectrootaccess)onthehostwhereyouareinstallingtheservices.Enoughfreespaceonthehosttoperformthedownloadandinstallation(minimum500MB,includingthedownload,but1GBto2GBispreferred).

WhatyouwillbeinstallingAttheendofthissection,youwillhaveafullyfunctionalChefserviceinstalledandreadytoworkwith.Beforeyougetstarted,let’slookatwhatyouwillbeinstallingonyoursystemsothatyouknowwhattoexpect.ThecomponentsthatmakeupaChefserviceareasfollows:

TheChefAPIserviceMessagequeue(AMQP)DatastorageSearchserviceWeb-basedmanagementconsole

TheChefAPIserviceisresponsiblefordeliveringrunlistsandreceivinginformationfromnodesaswellasprovidingawayforasystemadministratortoconfigurerecipes,runlists,databags,andthelike.Inordertogeneratethisdata,theAPIservicereliesonitspersistentdatastorageengine,inthiscasePostgreSQL,tostoreitsdata.TheoptiontosearchfordataisprovidedbytheSolrsearchengine,andRabbitMQisresponsibleforgluingthemalltogether.Together,thesecomponentsprovideChefwiththeabilitytodistribute,store,index,andmanageyourinfrastructure’sconfigurationdata.

GettingtheinstallerTheeasiestwaytoinstallChefisthroughasingledownloadablepackage,whichisprovidedforUbuntu10.04through12.10andRedHatEnterpriseLinuxVersions5and6.Thispackage,referredtoastheomnibusinstaller,containseverythingyouneedtogetaserverupandrunning.Youcanfinditonhttp://www.getchef.com/chef/install/.

Atthetimeofwritingthis,11.0.10isthelatestversionandistheonethatwillbeusedforthisbook.Thenewerversionofthe11.xseriesofChefshouldhaveaverysimilar,ifnotidentical,configuration.Notethattheseinstallersaresomewhatlarge,beingthattheycontainallofthedependenciesneeded.Forexample,theUbuntu12.10packageforChef11.0.10isapproximately200MBinsize.

TipAlthoughthesearetheofficiallysupporteddistributionsandreleases,itisentirelypossiblethattheseinstallerswillworkondifferentbutcompatibledistributions.Itmaybepossible,forexample,touseCentOSinsteadofRedHatEnterpriseLinuxorDebianinsteadofUbuntu.However,thesewillmostlikelyrequiresomemanualdependencyresolutionsandmaynotworkwithoutalotofeffort(andeventhen,possiblynotatall).

InstallationoutlineInstallationonallsupportedplatformsisrelativelysimilar.TheonlykeydifferencesarethenamesofthepackagefilesthatyouwilldownloadandthecommandsyouwillusetoinstallChef.

Thehigh-levelstepsyouwilltakeareasfollows:

1. DownloadingtheChefinstallerforyourplatform.2. Installingthepackageasanadministrativeuser.3. ConfiguringtheChefservice.4. Testingtheserverusingcommand-linetools.

Becausesteps3and4willbethesameforbothUbuntuandRedHatinstallationprocedures,theinstructionswillbeinasectionfollowingtheRedHatinstallationguide.

InstallingonUbuntuThefollowingareinstructionsforanUbuntu-basedsystem;theywereperformedonanUbuntu12.04host,butshouldbeidenticalforallsupportedUbuntudistributions.ForRedHat-basedinstallationinstructions,seethenextsection.

DownloadingthepackageYoucandownloadthepackagebyreturningtothedownloadpagereferencedpreviously(http://www.getchef.com/chef/install/),oryoucandownloadVersion11.0.10directlyfromhttps://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-1.ubuntu.12.04_amd64.deb.

InstallingthepackageInordertoperformtheinstallation,openaterminalonyourUbuntuhost(eitherlocallyorconnectviaSSH)asauserwhohasadministrativeprivileges.Thiscanbedonedirectlyeitherastherootoranyuserwhohaspermissiontoexecutearbitrarycommandsviasudo.

Onceyoulogintothehost,navigatetowhereyouwanttostorethepackage(rememberit’squitelarge,approximately200MB)anddownloadthefileusingcurl:

user@ubuntu:~$curl-Ohttps://opscode-omnibus-

packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.10-

1.ubuntu.12.04_amd64.deb

Oncethefileisdownloaded,thedpkgtoolwillbeusedtoperformthepackageinstallation:

user@ubuntu:~$sudodpkg-ichef-server_11.0.10-1.ubuntu.12.04_amd64.deb

Oncethisisfinished,theUbuntu-specificportionofthesetupiscomplete,andyouwillneedtoconfigureChefusingthechef-server-ctlcommand,whichwewilldiscussintheConfiguringChefServersection,followingtheInstallingonRedHatEnterpriseLinuxsection.

InstallingonRedHatEnterpriseLinuxInstallationonaRedHatEnterpriseLinuxdistributionisasstraightforwardasinstallinganyotherpackage.YoudownloadthepackagetothelocaldiskandinstallitusingRPMtools.

DownloadingthepackageYoucandownloadthelatestversionofthepackagebyreturningtothedownloadpagereferencedpreviously(http://www.getchef.com/chef/install/),oryoucandownloadVersion11.0.10directlyfromhttps://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm.

Inordertoperformtheinstallation,openaterminalonyourRedHathost(eitherlocallyorconnectviaSSH)asauserwhohasadministrativeprivileges.Thiscanbedonedirectlyeitherastherootoranyuserwhohaspermissiontoexecutearbitrarycommandsviasudo.

Onceyoulogintothehost,navigatetowhereyouwanttostorethepackage(rememberit’squitelarge,approximately200MB)anddownloadthefileusingcurl:

user@rhel:~$curl-Ohttps://opscode-omnibus-

packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm

Howlongthistakeswillvaryaccordingtotheavailablebandwidthbutshouldtakesomewherebetween5and20minutesonareasonablyfastconnection.

Oncethefileisdownloaded,therpmtoolwillbeusedtoperformthepackageinstallation:

user@rhel:~$sudorpm-ivhchef-server-11.0.10-1.el6.x86_64.rpm

Oncethisisfinished,theRedHat-specificportionofthesetupiscomplete,andyouwillneedtoconfigureChefusingthechef-server-ctlcommand,whichwewilldiscussinthefollowingsection.

ConfiguringaChefserverHistorically,installingChefrequiresmanualeditingofconfigurationfiles,choosingRabbitMQcredentials,installingCouchDB,andahandfulofothertasks.Now,withtheomnibusinstaller,allofthisistakencareofforyou.Ifyouhavebeenfollowingalong,yoursystemhastheChefserverandallofitsdependenciesinstalledonthesysteminthe/opt/chef-serverdirectory.

Includedwiththeinstallationofthepackageisashellscript,chef-server-ctl(locatedat/opt/chef-server/bin),whichisresponsibleforconfiguringyournewlyinstalledChefserver.Inordertoconfigureyourservices,youwillneedtorunitasrootbecausethescriptswillneedtomodifyyoursysteminwaysthatyourregularaccountmaynotbeableto.Initializingtheconfigurationtoolisassimpleasissuingthefollowingcommand:

sudochef-server-ctlreconfigure

Runningthisscriptmaytakeafewminutes,anditwillproducealotofoutputwhileitisdoingitswork.Whileitisrunning,let’stakeafewminutestodiscusshowitworksandwhatitisdoing.

Understandinghowchef-server-ctlworksEarlierinthischapter,youwerebrieflyintroducedtothechef-solotool.Yousawhowitcanbeusedtomanageyourlocalserverusingon-diskrecipesandconfigurationdata.TheChefteamhasleveragedthisabilitytodojustthatwiththeChefserverusingchef-solotobootstraptheserverconfiguration.Ifyouweretolookatthecodeforthe/opt/chef-server/bin/chef-server-ctlscript,youwouldseethatthelastlineinthescriptexecutesthefollowingcommand:

/opt/chef-server/embedded/bin/omnibus-ctlchef-server/opt/chef-

server/embedded/service/omnibus-ctl$@

Ifyoufollowthetrailanddigintotheomnibus-ctlscript,youwillfindthatitisjustawrapperaroundtheomnibus-ctlRubygem.Diggingintotheomnibus-ctlgem,youwillseethatintheend,thereconfigurecommandyoupassonthecommandlineisaRubymethodthatmakesthefollowingcall:

run_command("chef-solo-c#{base_path}/embedded/cookbooks/solo.rb-j#

{base_path}/embedded/cookbooks/dna.json")

ThistellsusthattheChefomnibuspackageuseschef-solotoconfigureitself—aprettyclevertrickindeed!Youcanseejusthowpowerfulatoolchef-solocanbe,beingabletoconfigureandreconfiguretheChefservice.

What’shappeningonmyserver?Whatyouwillprobablynoticerightawayisthatalotoftextisbeingscrolledpastinyourterminalwindow.Ifyouweretolookatthecontents,youwouldseethatitshowsyoutheactionsthatarebeingtakenbychef-solotoprovisionyournewservices.Asthereisalotofinformationgoingpast(thousandsoflines),hereisahigh-leveloverviewofwhatishappeningonyourhost:

1. Anewuser,chef_server,anditscorrespondinggrouparebeingprovisioned.2. Chefservicesarebeingsetup,andstartupscriptsforupstartarebeingplacedinthe

appropriatesystemdirectories.TheRunscriptsforChefservicesarelocatedat/opt/chef-server/sv.

3. Chefstatedirectoriesarebeingcreatedin/varincluding/var/opt/chef-serverand/var/log/chef-server.

4. RabbitMQisbeingconfiguredtostoredatain/var/opt/chef-serverandlogtheoutputto/var/log/chef-serveraswellasitsstartupscriptsin/opt/chef-server/sv/rabbitmq/run.

5. PostgreSQLisbeingconfiguredwithitsdatain/var/opt/chef-server/postgresql/dataalongwithauser,opscode-pgsql,toruntheservice.Somesystem-levelchangestosharememorysizesarebeingsetviasysctltomakePostgreSQLworkaswellaspersistedinsystctl.conf.

6. Solrisbeingsetuptoworkwiththeconfigurationanddatarootedin/var/opt/chef-server/chef-solr/,withtherunscriptbeingplacedin/opt/chef-server/sv/chef-solr/run.

7. Chef-expander(thedata-indexingservice)isbeingconfiguredfor/var/opt/chef-server/chef-expanderasitsworkingdirectorywithSolrandRabbitMQendpointsonthelocalhost.Therunscriptislocatedat/opt/chef-server/sv/chef-expander/run.

8. TheChefbookshelfmetadataserviceisreadiedin/var/opt/chef-server/bookshelf/withitsrunscriptat/opt/chef-server/sv/bookshelf/run.

9. Erchef,theErlangChefservice,isinstalledandpointedatthelocalSolr,RabbitMQ,bookshelf,andPostgreSQLservices.

10. Thesystemisthenbootstrappedusingthebootstraprecipe.Thisrecipeverifiesthatthesystemisrunning(bycheckingthatthehttp://localhost:8000/_statusreturnsanHTTP200response)andinstallstheSSLcertificatefortheweb-basedUIin/etc/chef-server/chef-webui.pem.

11. Theweb-basedUIconfigurationfilesaregeneratedandplacedin/var/opt/chef-server/chef-server-webui/.

12. AcopyofnginxtohostthewebUIisplacedin/var/opt/chef-server/nginx,andtheinitialself-signedSSLcertificatesaswellasthestaticassetsareinstalledin/var/opt/chef-server/nginx/html.

13. TheChefAPItestingframework,chef-pedant,isinstalled.14. Finally,/etc/chef-server/chef-server-running.jsonisgeneratedwiththe

currentconfigurationsettingsforyourChefservices.

Clearly,thereisalothappeninghere;ifyouhaveanyoutstandingconcernsaboutwhatisbeingdone,besuretoreadthroughtheoutput.OneofthegreatthingsaboutChefisthattherecipesarejustasetofscriptsthatyoucanopenandviewthecontentsof,andtheoutputshowsyouwhatishappeningduringtheexecution.Everythingitdoesistransparentandmanageablebyyou.

VerifyingthattheservicesarerunningOncetheconfigurationofyourservicesiscomplete,youwillwanttovalidatethattherequiredservicesarerunning.Again,thechef-server-ctlscriptwillbeused,butwewillinvokethestatussubcommandinsteadofthereconfiguresubcommand,asshowninthefollowingcode:

user@host:~$sudochef-server-ctlstatus

run:bookshelf:(pid3901)3123s;run:log:(pid3900)3123s

run:chef-expander:(pid3861)3129s;run:log:(pid3860)3129s

run:chef-server-webui:(pid4053)3095s;run:log:(pid4052)3095s

run:chef-solr:(pid3819)3135s;run:log:(pid3818)3135s

run:erchef:(pid4230)3062s;run:log:(pid3937)3117s

run:nginx:(pid4214)3064s;run:log:(pid4213)3064s

run:postgresql:(pid3729)3146s;run:log:(pid3728)3146s

run:rabbitmq:(pid3423)3172s;run:log:(pid3422)3172s

ThestatussubcommandwillshowyoutheprocessIDofeachcomponent,howlongithasbeenrunningfor,thePIDoftheloggingprocessassociatedwiththatservice,andhowlongtheloggingservicehasbeenrunning.Forexample,wecanseethatchef-server-webuihasaPIDof4053andhasbeenrunningforclosetoanhour,andtheloggerhasaPIDof4052,havingbeenrunningforjustaslongastheservice.

Asyoucansee,theinstallationofChefyieldsanumberofcomponentsthatwillneedtobeupandrunninginordertosuccessfullyuseChef.Youshouldhavethefollowingcomponentsrunningandlisteningonthefollowingnetworkports:

Component Whattolookforintheprocesslist Port(s) Public?

ChefAPIserver Erchefandnginx 80,443 Yes

Webmanagementconsole chef-server-webuiandnginx 80,443 Yes

Dataindexer chef-expander N/A N/A

Solr java(runningstart.jarintheChefdirectory) 8,983 No

PostgreSQL postmaster 5,432 No

RabbitMQ beam.smprunningrabbit 5,672 No

Publiccomponentsneedtobemadeavailabletoanyclients,nodes,orendusersthatexpecttousetheChefserviceoverthenetwork.Configuringyourinfrastructuretoensurethatyourservicesareavailableviathenetworkisoutsideofthescopeofthisbookasthereareanear-infinitenumberofpossiblenetworkconfigurations.

Atahigherlevel,makesurethatanyfirewalldevicesorpacket-filteringsystemsarenotpreventingtrafficfromreachingtheseservicesifyouseethattheyarerunning,butarehavingdifficultiesinconnectingtothem.Ifanyoftheseservicesarenotrunning,youwillneedtoconsultthelogfilesgeneratedbytheservicetodeterminewhatmightbepreventingthemfromstartingup.

ValidatingthatyourserviceisworkingInordertoworkwithChef,youwillneedawaytointeractwithit.Fortunately,Chefprovidesasuiteofcommand-lineutilities,whichwewilldiscussatlengthasthebookprogresses.Thereisoneprimarytool,knife,thatallowsanadministratortointeractwiththeserviceinthecommandline.Theknifetoolisrunfromaworkstationandprovidesmanycommandstoview,search,andmodifydatamaintainedbytheChefservice.Onceyouhaveinstalledandverifiedthatalltheservicesarerunning,wecanmoveontosettingupknife.

TipYouwillseethatthestandardplacetostoreyourChefconfigurationdataisin$HOME/.chef(onaUNIX-likesystem.)Thisisnotmandatory,andthesefilescanbestoredanywhereyoulike.

TheknifetoolcommunicateswiththeChefserverviaHTTPandusescertificatesforauthenticationbetweentheworkstationandtheserver.Inordertogetstartedwithknife,wewillneedtodotwothings:gainaccesstothecertificatesthatweregeneratedduringtheinstallationofChefandthenusethosecredentialstosetupanewuserinthesystem.

Inthefollowingexamples,wewillbeusingthehostthattheChefserviceswereinstalledonasourworkstation(wherewewilluseknife).Ifyouwanttouseadifferenthost,youwillneedtogettherequiredcertificate(.pem)filestoyourlocalmachineusingscporsomeothermechanism.Byusingthefollowingcommands,wecangettherequiredauthenticationmaterialsintoourworkdirectory:

mkdir$HOME/.chef

sudocp/etc/chef-server/admin.pem$HOME/.chef

sudocp/etc/chef-server/chef-validator.pem$HOME/.chef

sudocp/etc/chef-server/chef-webui.pem$HOME/.chef

sudochown–R$UID$HOME/.chef

TipChefusesasignedheaderauthenticationforrequeststotheAPI,whichmeanstheremustbeasharedkeythatispresentonboththeclientandtheserver.Chef-serverwillgeneratethechef-validator.pemfilewhenitisconfigured.Newnodesorclientsusethechef-validator.pemfiletosigntherequestsusedtoregisterthemselveswiththesystem.

OnceyouhavethesefilescopiedintoyourChefworkdirectory,itistimetoconfigureknifeitself.Fortunately,knifehasaninteractiveconfigurationmodethatwillwalkyouthroughtheprocessofgeneratingaconfigurationfile.First,ensurethatyouareusingyourChefgemset(ifyouareusingRVMaswediscussedearlier)andthenrunknifeonyourworkstation(again,inthisexample,weareusingourChefservicehostforbothpurposes):

user@chef:~$rvmuse1.9.3@chef

user@chef:~$knifeconfigure-i

Whenyourunknifewiththe-iflag,youwillbepromptedbythefollowingquestions,

whichyoucananswerwiththedefaultsforalmosteverything(non-defaultanswersareinbold):

WARNING:Noknifeconfigurationfilefound

WhereshouldIputtheconfigfile?[/home/user/.chef/knife.rb]

PleaseenterthechefserverURL:[https://localhost:443]

Pleaseenteranameforthenewuser:[user]

Pleaseentertheexistingadminname:[admin]

Pleaseenterthelocationoftheexistingadmin'sprivatekey:[/etc/chef-

server/admin.pem]~/.chef/admin.pem

Pleaseenterthevalidationclientname:[chef-validator]

Pleaseenterthelocationofthevalidationkey:[/etc/chef-server/chef-

validator.pem]~/.chef/chef-validator.pem

Pleaseenterthepathtoachefrepository(orleaveblank):

CreatinginitialAPIuser…

Pleaseenterapasswordforthenewuser:

Createduser[user]

Configurationfilewrittento/home/user/.chef/knife.rb

user@chef:~$

Asmentionedearlier,thisdoestwothings:

First,itusesthevalidationkeyandclientnamespecifiedatthepromptstocontacttheAPIserviceandregisteranewclient(user)withtheserviceSecondly,itgeneratesaconfigurationfileforknifethathasthesettingsneededtoconnecttotheservicefromnowon

SinceChefanditscomponentsarewritteninRuby,theresultingconfigurationfileisaRubyscript,whichcontainssomecodethatconfiguresknifesothatitknowswhatAPIservertoconnectto,whichkeyfilestouse,whatclientnametouse,andsoon.

Aninspectionoftheconfigurationfilethatwasgeneratedbythepreviouscommandwilllooklikethefollowing:

log_level:info

log_locationSTDOUT

node_name'user'

client_key'/home/user/.chef/user.pem'

validation_client_name'chef-validator'

validation_key'/home/user/.chef/chef-validator.pem'

chef_server_url'https://localhost:443'

syntax_check_cache_path'/home/user/.chef/syntax_check_cache'

Becauseweareusingtheservicehostasourworkstation,theChefserverURLpointstothelocalhost.Ifyourworkstationweretobeadifferentsystemsuchasyourlaptop,thenthisURLwouldbetheIPorhostnameofthehostrunningtheChefservice.

EnsuringthatyourknifeconfigurationworksAftersettingupknife,wecanuseittovalidatethatitwasconfiguredcorrectlybyqueryingtheChefserverusingsomesimplecommands.Theknifecommandsfollowtheformatknife<command><subcommand>,wherecommandiseitheraclient,configuration,cookbook,cookbooksite,databag,environment,exec,help,index,node,recipe,role,search,ssh,status,ortag.Subcommandswillvarywiththecommand,buttheytypicallyincludethingssuchasshow,create,list,anddelete(amongothers).

Astherewillinitiallybenonodes,cookbooks,recipes,roles,databags,andsuch,wewillquerythelistofclientsthattheserverknowsabout.Thisshouldbealistoftwoclients:chef-webui(asitisaconsumeroftheAPIitself)andchef-validator(withoutit,itwouldn’tbepossibletoregisteranewclient).

Theclientcommand,withthelistsubcommand,printsalistofclientsthattheserverknowsabout.Atthispoint,runningthecommandwouldlooklikethis:

user@chef:~$knifeclientlist

chef-validator

chef-webui

user@chef:~$

TipIfyoudonotgetthepreviousoutput,butgetanerrorinstead,youwillneedtogobackandmakesurethatallthepreviousstepsarecompletedandverified.

Onceyouknowthatitworks,youcanuseknifetointeractwiththeAPI.Unfortunately,wedonothavemuchdatainthesystemjustyet,butwecanusetheshowsubcommandinconjunctionwiththeclientcommandandaclientnametodisplaymoredetailedinformationaboutaclient:

user@chef:~$knifeclientshowchef-webui

admin:true

chef_type:client

json_class:Chef::ApiClient

name:chef-webui

public_key:-----BEGINPUBLICKEY-----

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAos5cQ1NxP7zKf1zRM33g

YeVyHNOO5NcICjSIvqQ5A37wwLfgtPLJQqboW7ZcNL3xYcKOlfYSEK7xha3ss8tT

A+XMifaFp3JsdheyPeIJir2bc9iltUUcbpw9PJ2aQKTBlFNx23A7ag+zBfxcDjbY

7RkdcziwB74ynd6e/K8c0JTRnA5NxoHkFc6v8a/itwujGwugWJXDQunWfCmAvjws

JgDOUu2aHOCVIVkc8it51Sc7Anx0YnCjNmdhz1xIo0MOVNOEmC9ypP0Z7mVv1C69

WWBOEvS9zimjXo4rxBwFmWkPEIG6yPQjhuNmFd69K14vZQtAsH07AZFRSS7HLWnZ

WQIDAQAB

-----ENDPUBLICKEY-----

validator:false

user@chef:~$

SummaryCongratulations!Ifyouhavegottenthisfar,younowhaveafullyfunctionalChefserviceandacopyofthecommand-lineutilities,includingchef-solo.Younowhavecoveredthefollowing:

UsingRVMInstallingchef-soloCreatingasimplerecipeRunningrecipeswithchef-soloInstallingtheChefserviceGettingstartedwiththeknifeutilityVerifyingthatyourChefserviceisoperatingcorrectly

NowthatyouareabletouseyourChefservice,wecanbegintoinvestigatehowtomodelourenvironmentwithChefandseewhatitcandoforus.

Chapter2.ModelingYourInfrastructureNowthatyouhavesetupyourserverorareusingthehostedoffering,let’sdiscusshowtomodelyourinfrastructurewithChef.Chefallowsyoutodothisusingbuildingblocksthatshouldbefamiliartoanysystemadministrator.

ThischapterwillcoverhowtomodelyourinfrastructurewithChef.Thiswillinvolvethefollowing:

LearningsometerminologiesandconceptsrelevanttoChefAnalyzingasimpleapplicationinfrastructureandseeinghowitcanbemodeledwithChefDecomposingourarchitectureintothevariouscomponentstobemodeledExamininghowdataisstoredandconfigurationsaregeneratedwithChefBootstrappingandprovisioninghostswithcloudproviderssuchasAWSandRackspaceCloud

GettingtoknowChefAswithanyothertoolorsystem,therearenewconceptsandterminologiestobelearned.Herearesometermsthatyoumayhaveseenorwillseeinthischapter:

Node:AnodeisasystemthatismanagedbyChef.Thesecanbeservers,desktopsystems,routers,andanythingelsethatiscapableofrunningtheChefclientandhasasupportedoperatingsystem.Workstation:AworkstationisaspecialnodethatisusedbyasystemadministratortointeractwiththeChefserverandwithnodes.Thisiswherethecommand-linetoolsareexecuted,specificallytheknifecommand-linetool.Bootstrap:ThisisaprocessofsettingupanodetobeusedasaChefclient.ThisinvolvesperforminganyworkrequiredtoinstallthedependenciesforChefaswellasChefitself.BootstrapScript:ThereareanumberofpossiblewaystoinstallChef,Ruby,othercorerequirements,aswellasanyadditionalconfigurationthatisneededforyourspecificsystems.Toprovidethislevelofflexibility,thebootstrapprocessisscripted;onWindows,thisisabatchfile.Recipe:Recipesprovidetheinstructionsrequiredtoachieveagoal,suchasinstallingasoftwarepackage,configuringafirewall,provisioningusers,installingprinters,ormanagingothersystemresources.ThesearewritteninRubyandexecutedonthenodesspecifiedbythesystemadministratorthroughtheChefconsole.Cookbook:Acookbookisacollectionofrecipes;typically,acookbookprovidesonespecificgroupofactionssuchasinstallingApacheorMySQL,providingChefresourcesforaspecificsoftwaretool,andsoon.Attributes:Variouscomponentsofthesystemhavetheirownattributesandpropertiesthatdescribehowthesoftwareistobeconfigured.Thesepropertiesaredefinedatvariouslevels,rangingfromnode-specificsettingstogeneraldefaultsforacookbookorarole.Role:Aroleisacollectionofrecipesandconfigurationdatathatdescribehowaresourceshouldbeconfiguredinordertoplaythatroleinyouroverallsystemarchitecture.ExamplesofrolesmightincludeMSSQLServers,ExchangeServers,IISServers,fileservers,andsoon.Aroledoesnotcontainanyknowledgeofresources(systems)toapplytheroleto,onlytheconfigurationdata.RunList:Arunlistisalistofrecipestobeappliedtoagivennodeinacertainorder.Arunlistcanbecomposedofzeroormorerolesorrecipes,andtheorderisimportantastherunlist’sitemsareexecutedintheorderspecified.Therefore,ifonerecipeisdependentupontheexecutionofanother,youneedtoensurethattheyruninthecorrectorder.Resource:Resourcesareawayofdescribingwhatarecipeisperforming.Someexamplesofresourceswouldincludefiles,directories,printers,users,packages,andsoforth.Aresourceisanabstractionofsomethingthatisconcretelyimplementedinaprovider.Provider:Aproviderisaconcreteimplementationofaresource.Forexample,auser

isagenericresource,butLDAPusersorActiveDirectoryusersareconcreteimplementationsofauserresource.Thetypeofproviderbeingselectedwilldependonsomefactors,suchastheplatform.Databags:Databagscontainshareddataaboutyourinfrastructure.Informationthatisnotspecifictoaroleoraresource,suchasfirewallrulesanduseraccounts,willbestoredindatabags.Thisisagoodplacetostoresystem-wideconfigurationdata.Environments:Environmentsprovidealevelofencapsulationforresources.Forexample,youmayhavetwoidenticalenvironments,onefortestingandoneforproduction.Eachofthesemayhavesimilarsetupsbutdifferentconfigurations,suchasdifferentIPaddressesandusers.

ModelingyourinfrastructureNowthatyou’remorefamiliarwithsomeofthetermsyouneedtoknow,let’stakealookatasamplemodelandmapittoChef’scomponents.Atahigherlevel,theapproachwewilltakeisasfollows:

1. Defineanoverviewofyourinfrastructurethatisdecomposedintorolestobeperformedwithinthemodel(webservers,firewalls,databaseservers,andsoon).

2. Collectordeveloprecipesthatdescribetheconfiguration,software,andactionstobeappliedforthoseroles.

3. BootstraphostswiththeChefclientsothattheycanparticipateinbeingmanaged.4. Addanyrequiredconfigurationdataintodatabagstobeusedbynodeswhenrunning

recipessuchasIPaddressranges,hostnames,users,softwarekeys,oranythingelsethatisspecifictotheactiveconfiguration.

5. Segregatehostsandconfigurationsintodifferentenvironmentstoprovideareplicatedinfrastructure(development,staging,production,andsoon).Thisstepisoptional.

Inthischapter,wewillbeusingCheftobuildtheinfrastructureforamulti-tiered,photo-sharingapplicationwhosecomponentsarediagrammedinthefollowingimage:

Buildinganarchitecturediagramgivesusagoodoverviewofoursystemsothatwecanhaveamapofthesystembeforewestartbuildingit.Itisimportanttonotethatamodelof

ourinfrastructuredoesn’tneedtobemappeddirectlytoresources(physicalorvirtual);rather,itprovidesanabstract,high-leveloverviewofyoursystems.Onceyouhavethemodel,youcanapplyittotheresourcesavailableasyouseefit.

Oursampleservice-orientedwebapplicationiscomposedofthefollowingsoftwareservices:

AfrontendwebapplicationserviceAnimage-processingengineAnimagesearchengine

Eachofthesecomponentsisarolethatisbeingplayedinthesystem.Theserolesmaycoexistonsharedresourcesormaybeappliedtodedicatedresources.Aservice-orientedarchitectureisagoodexampletoworkwithforseveralreasons:

ItisflexibleandscalableItwillprovideuswithacompletesystemthatiscomposedofmultipleindependentcomponentstomodel,makingitmoreinterestingasanexample

Inthisexample,inadditiontotheseroles,wemightwanttofurtherconfigureourinfrastructuretoprovidetwodifferentenvironments:oneforstagingandintegrationtestingandoneforproduction.Again,becausethisisamodel,ourstagingenvironmentandproductionenvironmentwillbecomposedofthesamerolesandhavethesameoverallarchitecture;however,eachwillhavedifferentresourcesandconfigurationdataassociatedwiththem.Youmaychoose,forexample,toconsolidateresourcesinatestenvironmentinordertokeepcostsdown.

Forthisinitialoverview,wewillassumethatwehaveanaccountwithapopularcloud-server-hostingcompany,thatthenetworkandoperatingsystemsareinstalledandoperational,andthatwehaveafunctionalandconfiguredChefserviceandworkstation.

Inourhypotheticalsystem,eachservicecanbemappedtoaspecificroleinChef.Tomodeltheinfrastructuredescribed,wewillhaveanumberofroles,oneperelementinourarchitecture.Inthiscase,wewillbuildoneroleforeachserviceinourstackastheyprovideveryspecificfeatures.

RolesAroledescribesapartthatasystemplaysinyourinfrastructurethroughacombinationofrecipestoexecuteandconfiguredata.Theserolescanbefine-grainedorbroadlydescribed,dependingonyourneeds.Therearebenefitsanddrawbackstoboththeapproaches:fine-grainedrolesaresmallerandeasiertoworkwithbutrequirealargernumberofrolestomanage,whereasbroadlyscopedrolesarelessflexibleandnotasreusable.

Forexample,consideratypicalLAMP(Linux,Apache,MySQL,andPHP)stack.Thestackcouldberepresentedbythreeroles:anApachewebservicewithPHP,aMySQLdatabaseservice,andanOpenSSHserviceforadministration.Alternatively,youcoulddefineonerolethatdescribestheinstallationoftheMySQLdatabaseservice,theSSHservice,andtheApacheservice.

Rolesthemselvesknownothingaboutresources;instead,theyareadescriptionofhowtoconfigureasysteminordertofillthatrole.Thesystemadministrator,viathechefconsole,assignsrolestothenode(s)thattheywillbeappliedto.Thismaybeaone-to-one,one-to-many,ormany-to-onemapping,dependinguponyourcapacityplanning.Atanytime,anadministratorcanchangethelistofrolesthatareappliedtoanode,addingorremovingthemasneeded.Forexample,youmightdecidetoapplyallyourrolestoonehosttodayforcostsavings,butscalethemoutinthefutureasyourbudgetandneedsgrow.

DefiningrolesLet’stakealookatsomeroleswemightdefinetomodelourSOAapplicationonasdescribedearlierinthechapter.Here,wewilldefinefine-grainedrolesastheyareeasiertodissectanddeployontoseparatenodeslater.Atahigherlevel,thefollowingrolesarewhatourservicesneedtoprovide.

Awebapplicationservicerole

Whendefiningwhatawebapplicationserverwillneedtodo,wewillneedthefollowing:

nginxHTTPserviceRuby2.0MemcachedservicePostgreSQLclientlibrariesOpenTCPportsontheexternalnetworks:80forHTTPand443forHTTPS

Animage-processingrole

Thisrolerequiressomeimage-processinglibrariesandcustomsoftwaretobeinstalled:

ImageMagicklibrariesGit(tocheckoutthesourcecode)Buildtools(tocompileoursource)Thelatestversionofourimage-processingsoftware

Animagesearchrole

Aservicethatprovidesimagesearchingthroughperceptualhashingwillprovideanimagesearchrolefunctionality.Thisrolewillrequirethefollowing:

AJavaruntimeenvironment(JREorJDK)Ourcustom-builtservicethatisdevelopedinJavaTCPport8999opentointernalhosts

APostgreSQLservicerole

ForthePostgreSQLdatabaseservicerole,thelistisasfollows:

PostgreSQL9.xserverTCPport5432opentointernalnetworkclientsDatabasebackupsoftwaretobackupdatatoanexternalclouddatastorageservicesuchasS3

ASolrservicerole

AsystemthatprovidestheApacheSolrservicewillneedthefollowing:

AcompatibleJavaruntime(OracleJREorOpenJDK)TCPport8993opentointernalserversApacheSolritself

AnOpenSSHservicerole

AnOpenSSHservicerolewillneedthefollowing:

OpenSSHserverTCPport22openonalltheinterfaces

Noticethattheseroleshavenospecifichostinformation,suchasIPaddressesorserverstoinstallthesoftwareonto;instead,theyareblueprintsforthepackagesweneedtoinstallandtheconfigurationthatthoseroleswillprovide,suchasopenports.Inorderfortheseroledefinitionstobemadeasreusableaspossible,wewillwriteourrecipestousenode-androle-specificconfigurationordatafromourdatabagstoprovidetherequiredconfigurationdata.

Inordertodefinetheseroles,youwillneedrecipesthatdescribethesetsofstepsandoperationsthatwillbeappliedtohostsinordertofulfilleachrole.Forexample,thePostgreSQLdatabaseserverwillrequireyoutoinstallPostgreSQL,openthefirewall,andsoon.Thesedefinitionsarecreatedbydevelopingrecipesthatcontainthenecessaryinformationtoperformthetasksrequired,suchasinstallingpackages,generatingconfigurationfiles,executingcommands,andsoon.Mostoftheservicesmentionedhere(ourcustomimagingsoftwarebeingthelikelyexception)havecookbooksthatalreadyexistandareavailablefordownload.

ImplementingaroleNowthatyouhaveseenwhatourinfrastructuremightlooklikeatahigherlevel,let’stakealookathowwewillgoaboutimplementingoneofourrolesinChef.Here,wewillimplementthePostgreSQLserverroleasitissimpletoconfigureandhasaveryrobustcookbookavailablealready.

Asmentionedbefore,youwillneedtoeitherdevelopyourowncookbooksordownloadexistingonesinordertobuildyoursystems.Fortunately,therearethousandsofcookbooksalreadywritten(over1,500asofthiswritingintheChefSupermarket)and,aswewillseeinfurtherchapters,developingnewcookbooksisastraightforwardprocess.

Inordertodefinearole,weneedtocreateit;thiscanbeaccomplishedthroughawebinterfaceorbyusingknife.Here,andelsewhereinthisbook,wewilluseknifeasthewaytointeractwiththeChefservicebecauseitprovidesaconsistentexperienceacrossself-managedandhostedChef.Solet’sgetstarted!

Thefirstthingyouwillneedtodoiscreateanewrolewithknife,whichisassimpleasexecutingthefollowing:

kniferolecreate-dpostgresql_server

ThiswilltellknifetoconnecttoyourChefserver’sAPIandcreateanewrolenamedpostgresql_server.The-dflagtellsknifetoskipopeninganeditorandinsteadacceptthedefaultvalues.IfyouwanttoseewhattheunderlyingJSONlookslike,omitthe-dflagandmakesureyouhaveanappropriateEDITORenvironmentvariableset.Onceyourunthis,youcanverifythatyourrolewascreatedwiththefollowingcommand:

kniferolelist

Thiswillshowyouthatyouhaveasingleroleinthesystem,postgresql_server.Currently,however,thisroleisemptyandhasnoinformationassociatedwithit,justanameandanentryinthesystem.Nowthatwehavearoleinthesystem,let’slookathowwecanworkwithsomerecipestomakeourroledosomethinguseful,suchasinstallthePostgreSQLservice.

DeterminingwhichrecipesyouneedRecipesarehowChefknowshowtomakesurethatthecorrectpackagesareinstalled,whatcommandsneedtobeexecutedinordertoopenportsonthefirewall,whichportsneedtobeopened,andsoon.Likeanygoodcook,Chefhasawidearrayofcookbooksatitsdisposal,eachofwhichcontainsrecipesrelevanttothatparticularsetoffunctionality.ThesecookbookscaneitherbedevelopedbythesystemadministratorordownloadedfromavarietyofplacessuchasGitHub,BitBucket,orfromacollectionofcookbooksmaintainedbytheChefcommunityontheChefSupermarket(http://supermarket.getchef.com).Wewilldiscusshowtodownloadandgetstartedwithsomesimplerecipesandthenfurtherdiscusshowtodevelopanddistributeourownrecipesinlaterchapters.

Consideringhowwehavearrangedourroles,wewouldneedrecipestoinstallandconfigurethefollowing:

nginxAPostgreSQLserverAPostgreSQLclientRuby2.0SolrJavaOpenSSHAMemcachedserverMemcachedclientlibrariesImageMagickGitACustomimagingsoftware(wewillcallitImage-O-Rama)

Here,wewilltakeanin-depthlookatthereciperequiredforourPostgreSQLserverandhowwecanleveragethattoinstalltheserviceonahost.

InstallingacookbookInstallingacookbookforuseonourclientsisquitesimpleandinvolvesonlytwosteps:

1. Developingacookbook,ordownloadingthecookbookfromsomewhere.2. UploadingthecookbooktotheChefserviceusingknife.

Togetstarted,wewilldownloadanexistingPostgreSQLcookbookfromtheChefcookbookcollectionanduploadittoourChefservice.NotethatinordertoinstallthePostgreSQLcookbook,youwillalsoneedtoinstallanydependenciesthatarerequired.Forsimplicity,theyareprovidedhereaspartoftheinstructions;however,youmayfindthatwhenyouexperimentwithothercookbooksinthefuture,youwillneedtodownloadafewcookbooksbeforeallofthedependenciesaremet,oruseatoolsuchasBerkshelfformanagingthem.

TodownloadacookbookfromChef’sprovidedcollectionofcookbooks,wewilluseknife

withthefollowingcommand:

knifecookbooksitedownload<cookbook_name>

Inthiscase,wewillneedtodownloadfivedifferentcookbooks:

postgresqlbuild-essentialaptchef-sugaropenssl

Foreachoftheitemsinthelist,wewilldownloadthemusingthefollowingcommand:

knifecookbooksitedownloadpostgresql

knifecookbooksitedownloadbuild-essential

knifecookbooksitedownloadapt

knifecookbooksitedownloadchef-sugar

knifecookbooksitedownloadopenssl

Eachdownloadwillresultinanarchivebeingdownloadedtoyourworkstation.Thesearchivescontainthecookbooks,andyouwillwanttodecompressthemafterdownloadingthem.Theycanbedownloadedanywhere,butitwouldprobablybeagoodideatokeeptheminacommoncookbooksdirectory,somethinglikechef/cookbooksinsideyourhomedirectorywouldbeagoodideaifyouneedone.

Oncetheyaredownloadedanddecompressed,youwillneedtouploadthemtotheChefservice.Thiscanbedonewithonlyonecommandusingknifecookbookuploadasfollows;theyareuploadedfromthedirectoryinwhichyoustoredyourdecompressedcookbooks:

knifecookbookupload-o.aptbuild-essentialpostgresqlchef-sugar

openssl

Thiswilluploadthefivecookbookswedownloadedandtellknifetosearchthecurrentdirectorybywayofthe-o.directive.Oncethisisdoneyoucanverifythattheyhavebeeninstalledusingtheknifecookbooklistcommand.

Oncetheyareinstalled,yourcookbooksareregisteredwiththeChefservice,andwecantakealookathowwecanconfigureandapplythePostgreSQLservertoanewUbuntuhost.

ApplyingrecipestorolesNowthatyouhavesomecookbooksregisteredwithyourChefservice,youneedtoaddthemtoarole’srunlistinorderfortheirbehaviortotakeeffectonanyendhosts.Therelationshipbetweenarecipeandanygivennodeisshowninthefollowingdiagram:

Becauseofthenatureofthisrelationship,recipesdeliberatelyhavenoknowledgeofindividualnodes.Justasarecipeforchocolatechipcookieshasnoideaaboutwhomanufacturedtherollingpinandspatula;aChefrecipeissimplyasetofinstructionsonwhattodoandinwhatordertoperformthoseactions.

Becausewehaveuploadedourcookbookstothesystem,wehavealreadyaddedtherecipescontainedinsideofthosecookbookstooursystem;therefore,wecannowassociatearecipewithourrecentlycreatedrole.Ifyoulookatthecontentsoftherecipesdirectoryinsideofthepostgresqlcookbook,youwillseethatthereisaserver.rbfile.ThisdescribesarecipetoinstallthePostgreSQLserverandiswhatwewillbeaddingtoourpostgresql_serverroleinordertoperformtheactualinstallation.

Todothis,weneedtoeditourroleandaddtherecipetoitsrunlist;wewilldothisusingknife.

TipEnsurethatyouhaveavalidtexteditorinyourEDITORenvironmentvariable;otherwise,youwillhavedifficultyeditingyourentitieswithknife.

Inordertoeditourrole,wecanusethekniferoleeditcommand:

kniferoleeditpostgresql_server

ThiswillopentheJSONfilethatrepresentsthepostgresql_serverrolestoredintheChefserverinatexteditorwhereyoushouldseethefollowingcontent:

"name":"postgresql_server",

"description":"",

"json_class":"Chef::Role",

"default_attributes":{

"override_attributes":{

"chef_type":"role",

"run_list":[

"env_run_lists":{

ThemostimportantsectionofthisJSONblobatthismomentistherun_listkey—thisisanarrayofallthethingswewanttorun.Thiscanbealistofrecipesorroles,andeachofthosehasthefollowingnamingstructure:

recipe[cookbook::recipe]forrecipesrole[role_name]forroles

Soourserverrecipeinsideourpostgresqlcookbookwouldthereforebenamed"recipe[postgresql::server]".Thisisexactlywhatwewillbeaddingtoourrole’srunlistJSON.Updatetherun_listentryfromtheoriginalvalueofanemptyarray:

"run_list":[

ToincludeourPostgreSQLserverrecipe,usethefollowingcode:

"run_list":[

"recipe[postgresql::server]"

ThisisallweneedtochangenowinordertoapplythePostgreSQLserverroletoournode.

TipNoticethatwehavenotaddedanyvaluestotherole’sattributes;thismeansthatourrecipewillbeexecutedusingitsdefaultattributes.Mostrecipesarewrittenwithsomesetofacceptabledefaultvalues,andthePostgreSQLserverrecipeisnodifferent.

Fornow,thereisnoneedtomodifyanythingelse,sosavetheJSONfileandexityoureditor.DoingthiswilltriggerknifetouploadyourmodifiedJSONinplaceofthepreviousJSONvalue(afterdoingsomevalidationonyourJSONcontents),andtherolewillnowhavethepostgresql::serverrecipeinitsrunlist.Youshouldseeanoutputfromknifeindicatingthattherolewassaved,andyoucanverifythatthisisthecasewithasimplekniferoleshowcommand:

kniferoleshowpostgresql_server

Thiswillshowyouanoverviewoftheroleinamorehuman-readableformatthanthesourceJSON.Forexample,ourroleshouldnowhaveoneentryintherunlistasshowninthefollowingoutput:

chef_type:role

default_attributes:

description:

env_run_lists:

json_class:Chef::Role

name:postgresql_server

override_attributes:

run_list:recipe[postgresql::server]

Oncethisiscomplete,ourroleisnowreadytobeappliedtooneofournodes.Atthispoint,wehaveuploadedourcookbooks,definedarole,andassociatedarecipewithournewlycreatedrole.Nowlet’stakealookatthefinalstep:applyingournewroletoanode.

MappingyourrolestonodesAshasbeendiscussed,rolesareadefinitionofwhatcomponentsneedtobebroughttogethertoserveaparticularpurpose;theyareindependentofthehardwaretheyareappliedto.Thishelpstoseparateconcernsandbuildreusablecomponentstoacceleratetheconfigurationofinfrastructureinnewarrangements.Inordertomanifestarole,itmusthaveanodethattheroleisappliedto;inordertomanageanode,itmusthavetheChefclientanditsdependenciesinstalledandberegisteredwiththeChefservice.

OnceanodeisregisteredwithChef,youcansetnode-specificproperties,assignrolesandrunthechef-clienttoolonthehostinordertoexecutethegeneratedrunlists.Foroursampleapplicationstack,wemayhavethefollowinghostsrunningUbuntuLinux14.04:

cayennewatermelonkiwi

OncetheyarebootstrappedandregisteredwiththeChefservice,wewillthendecidewhichrolesaretobeappliedtowhichnodes.Thiscouldyieldaconfigurationthatlookslikethefollowing:

cayenne

Webapplicationservicerole

watermelon

APostgreSQLdatabaseroleASolrsearchenginerole

Animage-processingroleAnimagesearchrole

Withoutanyhardware,rolesarejustanabstractblueprintforwhatneedstobeconfiguredtogethertoprovideaparticulartypeoffunctionality.Here,wehavecombinedourresources(cloudinstancesordedicatedhardware)andourrecipestobuildaconcreteinstanceofourservicesandsoftware.

Inordertoapplyournewlycreatedroletoourhost,watermelon,wewillneedtobootstrapthathost,whichwillinstalltheChefclientonthehostandthenregisteritwiththeChefservice.Thisisreallyasimpleprocess,aswewillseehere,andisachievedusingtheknifebootstrapcommand:

knifebootstrap-xroot-dubuntu14.04<ipaddress>

TipForourexample,thenodewilluseanUbuntu14.04hostcreatedonDigitalOcean,aninexpensivecloud-hostingprovider;youcanbootstrapjustaboutanymodernLinuxdistribution,butifyouarefollowingalongwiththecommandsinthebook,youwillget

thebestresultsbyusinganUbuntu14.04machine.

ThisprocesswillgothroughthestepsrequiredtoinstalltheChefclientonthenodeandregisteritwithyourChefservice.Onceitiscomplete,youwillseethattheChefclienthasfinishedwithanoutputsimilartothefollowing:

ChefClientfinished,0/0resourcesupdatedin4.264559367seconds

Ifyouwanttoverifythatthehosthasbeenadded,asimpleknifenodelistcommandwillshowyouthatithasbeenregisteredwiththeChefservice.Ifyoudon’tseetheclientoutputabove,oryoudon’tseethenewlybootstrappednodeinyourlist,makesurethattheoutputofknifebootstrapdoesn’tindicatethatanythingwentwrongalongtheway.

Onceournodeisregistered,wecanaddourpostgresql_serverroletoournode’srunlistusingthefollowingknifecommand:

knifenoderun_listsetwatermelonrole[postgresql_server]

Thiscommandwillsettherunlistonournewhost,watermelon,tocontainthepostgresql_serverroleasitsonlyentry.Thiscanbeverifiedusingtheknifenodeshowcommand:

knifenodeshowwatermelon

NodeName:watermelon

Environment:_default

FQDN:watermelon

IP:162.243.132.34

RunList:role[postgresql_server]

Roles:

Recipes:

Platform:ubuntu14.04

Nowthatthenodehasarunlistwithentries,it’stimetoactuallyconvergethenode.ConverginganodemeansthattheChefserverwillcompiletheconfigurationattributesandthenprovidetheendhostwithacompletelistofrecipestorunalongwiththerequiredcookbookdataandthenexecutethemonthenode.

ConverginganodeConverginganodeisdonebyexecutingthechef-clientcommand-lineutilityonthehost;thiscanbedoneinoneoftwodifferentways.ThesimplestwayistoSSHintothehostusinganSSHclientandthenexecutechef-clientastheroot;anotherwayistouseknifetoexecuteacommandonasetofhostsinparallel,whichwillbediscussedinlaterchapters.Fornow,simplySSHintoyourserverandrunchef-clientastheroot:

root@watermelon:~#chef-client

TheChefclientwillconnecttotheChefserviceanddownloadanyinformationneededtoexecuteitscompleterunlist.Anode’srunlistisdeterminedbyexpandingeveryentryinthenode’srunlistuntilitisalistofrecipestoexecute.Forexample,ournodecontainsoneelementinitsrunlist,thepostgresql_serverrole.Thisrole,inturn,containsoneentry,thepostgresql::serverrecipe,whichmeansthatthefullyexpandedrunlistforournodecontainsonlyoneentry.Inthissimplecase,wecouldhavejustaddedtherecipedirectlytoournode’srunlist.However,thishasanumberofshortcomings,includingnotbeingabletoaddextraconfigurationtoallthePostgreSQLserversinourinfrastructure,aswellasanumberofotherreasonsthatwillbediscussedinthefuture.

Inadditiontocomputingtherunlist,theChefservicewillalsodeterminewhatthefinalsetofconfigurationdataforournodewilllooklikeandthendeliverittotheclient.Thisiscomputedaccordingtoasetofrulesshownlaterinthischapter.Oncethatisdelivered,theclientwillalsodownloadallthecookbooksneededinordertoknowhowtoexecutetherecipesspecifiedinthefinalrunlist.Inourcase,itwilldownloadallthefivecookbooksthatweuploadedpreviously,andthen,whentheclientruniscomplete,theresultwillbepresentedinarunningPostgreSQLserver.

Oncetheclientruniscomplete,itwillreportonhowlongtheruntookandhowmanyresourcesweremodified.Theoutputshouldlooksomethinglikethefollowing:

ChefClientfinished,8/10resourcesupdatedin61.524995797seconds

Assumingthatnothingwentwrong,youshouldnowhaveafullyfunctionalPostgreSQLserverdeployedtoyournewhost.ThiscanbeverifiedbylookingattheprocesslistforaPostgreSQLprocess:

root@watermelon:~#psax|greppost

11994?S0:00/usr/lib/postgresql/9.3/bin/postgres-D

Thereyouhaveit,withonlyonecommand;yournodehasnowbeenprovisionedasaPostgreSQLdatabaseserver.Nowlet’stakealookathowwecanusesomeotherfeaturesofCheftomodelourinfrastructure.

EnvironmentsBeyondcreatingrolesandhavingresourcestoapplythemto,thereareoftenrequirementsaroundgroupingcertainresourcestogethertocreateadistinctenvironment.Anexampleofthismightincludebuildingastagingenvironmentthatfunctionsexactlylikeaproductionenvironmentforthepurposesofpreproductiontestingandsimulation.Inthesecases,wewouldhaveanidenticalsetofrolesbutwouldverylikelybeapplyingthemtoadifferentsetofnodesandconfigurationvalues.Chefachievesthisseparationthroughtheenvironmentprimitive,whichprovidesawaytogroupseparateconfigurationsandhoststogethersothatyoucanmodelsimilar,yetseparate,portionsofyourinfrastructure.

Inourhypotheticalinfrastructure,thethreehostsinourproductionenvironmentmaybecondenseddowntooneserverinpreproductioninordertosavemoneyonresources(orforotherreasons).Todothis,wewouldneedtobootstrapanode,perhapsnamedpassionfruitandthenconfigureittohavealloftherolesappliedtoit,ratherthanspreadingthemoutacrosssystems,asshowninthefollowingfigure:

Here,inthepreviousimage,youcanseethateachenvironmenthasaverysimilarsetupbutadifferentsetofIPaddressesandresources.Eventhoughwehaveaheterogeneoushardwarescaleinourenvironments(productionhasthreenodes,andpreproductionhasonlyone),anychangeswemakewillbeappliedtoallofthesystemsinaconsistentmanner.

Inordertoachievethistypeoffunctionality,Chefneedsawaytoorganizeandcompiletheconfigurationdatainordertoprovideittotheendhostwhenthetimecomestoconfigurethehost.NowthatweunderstandhowtomodeloursystemswithChef,let’stakealookathowChefhandlestheconfigurationdatatomakeallofthishappen.

OrganizingyourconfigurationdataChefrunsonconfigurationdata—thisdatacanbestoredinavarietyofdifferentlocations,andwithavarietyofpurposes.Whencomputingthefinalconfigurationforagivennode,thesourcesofconfigurationdataarethen“squashed”intoasingle,authoritativeconfigurationtobedeployed.Thosesourcesincludethefollowing:

Cookbooks:ToprovidereasonabledefaultsforrecipesNodes:Node-leveloverridesanddefaultsRoles:Per-roleconfigurationdataDatabags:System-wideconfigurationdatastorage

Datafromtheselocationsiscombinedtoproduceafinalhashofattributeswhenaclientrequestsitsrunlistfromtheserver.Cookbooksprovideabaselinesetofattributesthattherecipesinsiderelyon.Theseattributesactas“sanedefaults”fortherecipesthat,intheabsenceofoverridingvalues,aresufficienttoexecutetherecipeswithoutextrawork.Othersources,includingtheenvironment,roleandnodeitself,mayinturnoverridetheseattributesinordertoprovidethefinalconfiguration.

Whendevelopingrecipes,theseattributescanbeaccessedthroughthenodehashandarecomputedbyChefusingasetofrulestodetermineprecedence.Theorderofprecedencewhencomputingthishashisbrokendownintothefollowinglevels(lowesttohighestpriority):

DefaultNormal(alsoset)Override

Withineachlevel,thesourcesofattributedatainorderofincreasingprecedenceareasfollows:

TheattributesfileinsideofacookbookEnvironmentRoleNode

Thismeansthatanode-specificoverrideattributetakesprecedenceoverallothers,whichinturnismoreimportantthantherole,environmentandcookbookoverrideattributes,andsodownthechainofprecedence.Asyourscopebecomesnarrowedfromthemostgenericdescriptionofarole—therecipes—tothemostspecificcomponentinthesystem—theactualnodeitself—thesesettingsoverridethemoregenericvalues.Anodeknowsbestwhattheauthoritativeconfigurationshouldbeasopposedtoarecipe,whichdoesnotknowanythingaboutresourcesonthehost.Forexample,considerthefollowingscenarioinwhichyouhavetwohosts,potassiumandchromium.Forsomelegacyreason,theirdisksareconfiguredslightlydifferently,asfollows:

Potassium:

16GBrootpartition

250GBSSDdatapartitionin/opt

Chromium:

32GBrootpartition400GBEBSdiskmountedat/usr

InordertoinstallthePostgreSQLdatabaseserver,youneedtomakesureyouinstallitatalocationthatprovidesenoughstoragespaceforthedata.Inthisexample,therewillbemoredatathaneitherrootdiskscancontain.Asaresult,thedatadirectorywillneedtoresidein/optonpotassiumand/usronchromium.ThereisnowaythatthePostgreSQLrecipecanaccountforthis,andthepostgresql_serverrecipedoesnotknowanythingaboutitsresources.Subsequently,thelogicalplacetoconfigurethedatadirectoryisatthenodelevel.Ifthedefaultlocationaccordingtotherecipewere/usr/local,thenanode-leveloverridemaynotbeneededforchromium;however,inthecaseofpotassium,itcouldbedirectedtostoredatain/opt/datainstead.

Whatallthismeansisthatasyoudeveloprecipes,anydefaultattributesetbyyourcookbookwillbethelowestpriority.Youcansafelysetsomereasonabledefaultsinyourcookbookknowingthattheywillonlybeusedasafallbackifnobodyoverridesthemfurtherdownthechain.

ExampleattributedataAsimpledefaultattributesfileforPostgreSQLcookbookmightlooklikethefollowing:

default['postgresql']['port']="5432"

default['postgresql']['data_dir']="/usr/local/pg/data"

default['postgresql']['bind_address']="127.0.0.1"

NoticethattheattributesforacookbookareaRubyhash.Typically,goodpracticedictatesthatthenamespace(firstkeyinthehash)isthesamenameasthecookbook(inthiscase,postgresql),butthisdoesnotneedtobethecase.Duetocookbooksoftencontainingmultiplerecipes,acookbook’sattributesfilewilloftenhaveper-recipedefaultconfigurations.ConsiderafurtherevolutionofthePostgreSQLattributesfileifitweretocontainrecipesforboththeserverandtheclientinstallation:

default[:postgresql][:client][:use_ssl]=true

default[:postgresql][:server][:port]="5432"

default[:postgresql][:server][:log_dir]="/var/log/pglog"

Therearetimeswhenjustasimpleattributesfiledoesn’tmakesensebecausetheconfigurationmaybedependentonsomepropertyofthenodebeingmanaged.ThefactthattheattributesfileisjustaRubyscriptallowsustoimplementsomelogicinsideourconfiguration(thoughyoushouldavoidbeingoverlyclever).Considerarecipewherethedefaultgroupfortherootuserdependsontheplatformyouareusing:"userdonBSDs,"nBSDsonUbuntuLinux,and"nUbuelsewhere.Chefprovidesamethod,value_for_platform,thatallowstheattributetobechangeddependingontheplatformtherecipeisbeingexecutedon,asthefollowingexampledemonstrates:

default[:users][:root][:primary_group]=value_for_platform(

:openbsd=>{:default=>"wheel"},

:freebsd=>{:default=>"wheel"},

:ubuntu=>{:default=>"admin"},

:default=>"root"

Whereitmakessense,attributescanalsobesharedbetweencookbooks.Therearelimitedusesforthis,anditshouldbeusedwithcareasitblurstheboundariesbetweencookbooksandcausesthemtobecometootightlycoupledwithoneanother.

DatabagsTherearetimeswhenconfigurationdatatranscendsarecipe,role,environment,ornode.Thistypeofconfigurationdatatendstobesystem-widedatasuchasthefollowing:

FirewallrulesforvarioustypesofhostsUseraccountsSSHkeysIPaddresslists(whitelistsandblacklists)APIkeysSiteconfigurationdataAnythingthatisnotuniquetoaspecificentityinyourinfrastructure

Databagsareveryfree-form,asthenameimplies;recipesthatrelyondatafromdatabagswillimposetheirownexpectationsoftheorganizationwithinadatabag,butChefitselfdoesnot.Databagscanbeconsidered,likeallotherChefconfigurationdata,tobeonelargehashofconfigurationdatathatisaccessibletoalltherecipesacrossallthenodesinthesystem.

KnowingwhentousedatabagsBuildingfirewallrulesareagoodusecasefordatabags.Agoodcookbookisanislandbyitself;itmakesasfewassumptionsabouttheworldaspossibleinordertobeasflexibleandusefulasitcanbe.Forexample,thePostgreSQLcookbookshouldnotconcernitselfwithfirewallrules,thatis,therealmofafirewallcookbook.Instead,anadministratorwouldleverageagenericfirewallmodelandacookbookwithaspecificfirewallimplementationsuchastheUFWcookbooktoprovidethosefeatures.Inthiscase,ifyouweretolookattheUFWcookbook,youwouldseetheufw::databagrecipemakinguseofdatabagstomakethefirewallrulesasflexibleaspossible.

TipUFWstandsforuncomplicatedfirewall,apopulariptables-basedfirewallrulegenerationpackageforLinuxthatcomeswithmanymoderndistributionsandeasesthemanagementofafirewallconfiguration.

Inthiscase,ufw::databagexpectsthatthereisaspecificdatabagnamedfirewallandinsideofitareitemsthatsharenameswithrolesornodes;thisisinlinewiththenotionthatdatabagsarefree-form,butcertaincookbooksexpectcertainstructure.Ifourinfrastructuremodelhadtworoles,web_server,anddatabase_server,thenourfirewalldatabagwouldhavecontainedtwoitemsnamedaccordingly.Theweb_serveritemcouldlooklikethefollowinghash:

"id":"web_server",

"rules":[{

"HTTP":{

"dest_port":"80",

"protocol":"tcp"

"HTTPS":{

"dest_port":"443",

"protocol":"tcp"

Here,idoftheitemmapstothenameoftheitem,whichisalsothenameoftherole,sothattheufw::databagrecipeknowswheretofetchthedataitneedstobuilditsinternalfirewallrules.Tocomputeitslistoffirewallrulestoapply,theufw::databagrecipeexaminesthelistofrolesthatthenodeisconfiguredwithandthenloadsthecorrespondingdatafromtheitemsinthefirewalldatabag.

Asyoucansee,databagsallowyoutostorecentralized,free-formconfigurationdatathatdoesnotnecessarilypertaintoaspecificnode,role,orrecipe.Byusingdatabags,cookbooksforconfiguringusers,databases,firewalls,orjustaboutanypieceofsoftwarethatneedsshareddatacanbenefitfromthedatastoredinadatabag.

Onemightwonderwhywehavedatabagswhenwealreadyhaveattributedata,andthatwouldbeagoodquestiontoask.Attributesrepresentthestateofanodeataparticularpointintime,meaningthattheyaretheresultofacompactionofattributedatathatisbeingsuppliedtoanodeatthetimetheclientisbeingexecuted.WhentheChefclientruns,theattributedataforallthecomponentscontributingtothenode’srunlistisevaluatedatthattime,flattenedaccordingtoaspecificprioritychain,andthenhandedtotheclient.Incontrast,databagscontainarbitrarydatathathasnoattachmenttoaspecificnode,role,orcookbook;itisfree-formdatathatcanbeusedfromanywhereforanypurpose.Onewouldnot,forexample,belikelytostoreuserconfigurationdatainacookbookoronaspecificnodebecausethatwouldn’tmakemuchsense;usersexistacrossnodes,roles,andevenenvironments.Thesamegoesforotherdatasuchasnetworktopologyinformation,credentials,andotherglobaldatathatwouldbesharedacrossafleet.

Large-scaleinfrastructureOneofthemanybenefitsofChefisthepowertoapplyrolestonodesatscale.Thismeansthatonceyoudefineasetofrolesandsomesupportingrecipes,youcanapplythemtoonehostjustaseasilyasanyother.TherearemanyorganizationsthatmanageverylargeinfrastructureusingChef,includingcompaniessuchasFacebook,Ancestry,andRiotGames.WithChef,configuringonehundredhostsisasstraightforwardasconfiguringonehost.Beingabletoachievescalabilitygoalswhileremainingcost-effectiveisacriticalpartofrunningatechnologybusiness.Tothisend,Chefprovidestoolstoautomatethecreation,provisioning,maintenanceandterminationofvirtualhostsusingtheprovidedtools,whichcanhelpachievebothscalabilityandconservationofresources.ThenextchapterdiscusseshowtouseCheftoextendyourinfrastructureintothecloud.

SummaryNowthatyou’velearnedthekeyterminologythatChefusesanddissectedanexampleinfrastructureabit,youcanseethefollowing:

Infrastructurecanbedecomposedintothevariousrolesthatresources(nodes)playwithinthatinfrastructureAcombinationofrecipesandconfigurationdataprovideuswithrolesthatdescribeapartofouroverallinfrastructureChefanalyzestherolesappliedtohardwareresources(nodes)andgeneratesarunlistthatisspecifictothenodethatthoserolesarebeingappliedtoArunlististhencombinedwiththecookbooks,recipes,templates,andconfigurationdatatobuildaspecificsetofscriptsthatareexecutedonthenodewhenthechef-clientisrunWecanapplythesemethodologiesofautomatedconfigurationtocloudserversandphysicalsystemsalike.

NowthatyouunderstandhowChefmodelsinteract,let’stakealookathowwecangetstartedwithcloudservicesusingChef.

Chapter3.IntegratingwiththeCloudBeingabletoconfigurenewhostsautomaticallymeansthatifyououtgrowyourexistingresources,youcaneasilybringupnewserverstoincreaseyourcapacitywithverylittleeffort.TheChefcommand-linetool,knife,providestheabilitytoprovisionnewhostswithcloudservicesautomaticallyfromthecommandlineifconfiguredcorrectly.ThischapterintroducesyoutousingChefwithtwopopularcloudplatformproviders:AmazonEC2andRackspaceCloud.Here,youwilllearntouseknifewithbothoftheseprovidersinordertoperformthefollowing:

ProvisionnewhostsaccordingtoyourhardwareneedsBootstraptheChefclientandregisterhostswiththeChefserviceListyourexistingcapacityTerminateunneededcapacity

Youwillseethatallofthiscanbedoneusingthecommand-linetoolsprovidedwithouteverhavingtologintotheprovider’swebinterface,andallofthiswithinafewminutesoftime.

LeveragingthecloudCloudcomputingproviderssuchasRackspaceCloudandAmazonEC2provideon-demandcomputingpoweratthepushofabutton,afeaturethathasbecomeimmenselypopularwithdevelopersandsystemsadministratorsalike.Oneofthemosttoutedbenefitsofcloudcomputingiscostsavings;however,theseon-demandinstancescanbecomeveryexpensiveiftheyareleftrunning.Oftenthecapacityoftimewillbeconfiguredinordertohandlelarge-scaleeventsandthenleftonlinebecauseofthetimerequiredtoreconfigurethesystemsiftheyareneededagain.Asunderutilizedcapacityendsupcostingmoneyratherthansavingit,beingabletoreduceorexpandthecapacityquicklyandeasilywillhelpyoumatchyourcomputingneedswhilesavingbothtimeandmoney.

Thissectionspecificallylooksattwoofthemorepopularcloudproviders:AmazonEC2andRackspaceCloud;however,thereareothers,andthetechniquesdescribedherewillbebroadlyapplicabletoanyothersupportedcloudprovider.

AmazonEC2AmazonEC2isaverypopularcloud-computingplatform,andknifehassupporttomanageEC2instancesfromthecommandlinethroughtheknife-ec2plugin.ThefollowingstepsdemonstratehowyoucanworkwithEC2:

1. InstalltheEC2knifeplugin.2. SetupyourSSHkeysforusewithEC2.3. ConfigureknifewithyourAWScredentials.4. FindthedesiredAMI.5. Provisionanewhostwithknife.6. Bootstrapthenewlycreatedhost.7. Configurethenewhostwitharole.

InstallingtheEC2knifepluginAsofChef0.10,theec2subcommandshavebeenmovedfrombeingbuiltinknifetoanexternalgem,knife-ec2.InordertouseEC2commands,youwillneedtoinstallthegem,whichcanbedoneviathefollowingcommand:

geminstallknife-ec2

ThiswillinstallallofthegemdependenciesthattheEC2pluginrequires.

TipSomeofthecloudproviderpluginshaveconflictingdependencies,soitmaybebesttoleverageagemmanagerinordertoisolatethem.Forexample,usingRVMorrbenv,youmightcreateoneRubygemenvironmentperprovidersothatyoucouldswitchbackandforthwithasimplecommandsuchasrvmgemsetusechef-ec2.

SettingupEC2authenticationInordertomanageyourEC2hosts,youwillneedyourEC2key-pairproperlyregisteredwithSSHandyourAWSaccesskeyssetinyourknifeconfigurationfile.

Todothefirst,makesureyouhaveyourEC2SSHkeysdownloadedandregisteredwithyourSSHagent.OnewaytodothisistoaddthefollowingtoyourSSHconfigurationfile,typically,$HOME/.ssh/config:

Host*.amazonaws.com

ForwardAgentyes

CheckHostIPno

StrictHostKeyCheckingno

UserKnownHostsFile=/dev/null

IdentityFile~/.ssh/ec2_keypair.pem

InordertoconfigureyourAWSkeys,youwillneedtoaddsomeinformationtoyourknife.rbconfigurationfile($HOME/.chef/knife.rb):

knife[:aws_access_key_id]="YOURACCESSKEY"

knife[:aws_secret_access_key]="SECRETKEY"

ThesekeystellknifewhichAWScredentialstousewhenmakingAPIcallstoperformactionssuchasprovisionnewhostsandterminateinstances.Withoutthis,knifewillbeunabletomakeAPIcallstoEC2.Withtheserequiredchangesmade,let’slookathowtocreateanewEC2instancewithknife.

ProvisioninganinstanceInitially,wewilllookatprovisioninganinstanceusingoneoftheUbuntuAMIs.Withknife,wecanspecifytheAMItouse,theavailabilityzonetotarget,andthesizeinstancetobecreated.Forexample,tocreateanm1.largesizeintheus-east-1eavailabilityzonewithUbuntu12.04.3LTS,wewouldneedtousetheAMIwithami-23447f4aasitsidentifier.

InordertodeterminetheAMIID,youwillneedtolookitupatthefollowingURL:

http://uec-images.ubuntu.com/

TipRememberthatwhendecidingwhichAMItouse,someoftheEC2instanceswillbe32bitandsome64bit;choosetheappropriateAMIbasedontheinstancetype,region,andstoragemethodyouwanttouse.

Theprogressofprovisioningcanbeseenusingthefollowingcommand:

$knifeec2servercreate-Iami-23447f4a-fm1.large-Zus-east-1e-N

<nodename>-xubuntu--sudo

Theoutputfromthepreviouscommandwillshowyoutheprogressoftheprovisioning(thismaytakeaminuteortwo,dependingontheregion,instancesize,howlongstatuscheckstake,andsoon):

[user]%knifeec2servercreate-Iami-23447f4a-fm1.large-Sec2-keypair

-Zus-east-1e–N<nodename>-xubuntu--sudo

InstanceID:i-0dfec92d

Flavor:m1.large

Image:ami-23447f4a

Region:us-east-1

AvailabilityZone:us-east-1e

SecurityGroups:default

Tags:Name:i-0dfec92d

SSHKey:ec2-keypair

Waitingforinstance…..............

PublicDNSName:ec2-54-80-59-97.compute-1.amazonaws.com

PublicIPAddress:54.80.59.97

PrivateDNSName:ip-10-157-31-234.ec2.internal

PrivateIPAddress:10.157.31.234

BootstrappingtheinstanceAsyoucansee,knifewilltellyouthepublicIPandpublicDNSnameofthenewinstancealongwiththeinstanceID,tags,andsoforth.Oncetheinstanceisprovisionedandisonline,itwillneedtobebootstrapped.Rememberthatbootstrappingwillinstallthe

ChefclientandregistertheinstancewiththeChefservice,whichwecandointhesamewaywebootstrapanyotherhost:

$knifebootstrap<instance-public-ip-address>-N<node-name>-xubuntu--

AsEC2provisionseachinstancewithanubuntuuserthathassudoprivileges,weprovidethebootstrapcommandwith–xubuntuand--sudotoensurewehavetherequiredprivilegestoperformthebootstrapping.Additionally,asyoumorethanlikelydonotwanttheAWS-providedDNSnameasthenodename,theChefnodenameissetthroughthe-N<node-name>commandlineflag.Oncethebootstrapstepisfinished,assumingthattherearenoerrors,verifythatyournewlyprovisionedhostislistedinyourchefservice:

$knifenodelist

TheoutputwillcontainyournewlybootstrappednodeID,asspecifiedbyyouinthecommandlineortheDNSname,ifyoudon’tspecifyanodename.YouhavenowprovisionedanewEC2instanceandregistereditwithyourChefservicewithonlytwocommands!

TerminatingtheinstanceOnceyouaredonewithtesting,youmaynotwanttoleavetheEC2instancerunning,asitwillincurcostsifitremainsidle.Toensurethisdoesn’thappen,performthefollowingfoursteps:

1. ListyourEC2instances2. DeletetheserverfromEC23. RemovetheserverfromChef4. VerifythattheinstancenolongerexistsinCheforEC2

TolistourEC2instances,usetheserverlistsubcommandoftheec2command,whichwilllistalloftheEC2instancesinthespecifiedregion.Ifyoudonotspecifyaregion,us-east-1isthedefaultregion.ThefullcommandtolistEC2serversisasfollows:

$knifeec2serverlist

Asanexample,executingthiscommandafterprovisioningthefirsthostwillshowatableofoneinstanceasfollows:

InstanceIDNamePublicIPPrivateIP

i-0dfec92di-0dfec92d54.80.59.9710.157.31.234

Formostknifecommands,youwillneedtheinstanceIDsotheprevioustablecanbetruncatedtofitinprint.

TipListingEC2nodeswillresultinatablethatcontainsallthecurrentlyprovisionedEC2instancesintheregionbymeansoftheEC2API,whichisseparatefromtheChefserviceAPI.ThismeansyouwillgetalistofalltheinstancesinEC2whetherornottheyare

registeredwithChef.ThefulltablewillcontainmostoftheinformationyoucanseeontheEC2controlpanel,includingthepublicandprivateIP,flavor,AMI,SSHkey,andsoon.

Deletinganinstanceisjustaseasyascreatingorlistingthem.Here,theserverdeletesubcommandisinvokedwiththeinstanceidentifiertobeterminated.ThiswillusetheEC2APItoissueaterminatecommand—thisisnotreversibleandsothecommandwillpromptyoutoensurethatyoureallydidwanttodeletetheinstance:

[user]%knifeec2serverdeletei-0dfec92d

InstanceID:i-0dfec92d

Flavor:m1.large

Image:ami-23447f4a

Region:us-east-1

AvailabilityZone:us-east-1e

SecurityGroups:default

SSHKey:ec2-keypair

RootDeviceType:instance-store

PublicDNSName:ec2-54-80-59-97.compute-1.amazonaws.com

PrivateDNSName:ip-10-157-31-234.ec2.internal

Doyoureallywanttodeletethisserver?(Y/N)

WARNING:Deletedserveri-0dfec92d

WARNING:Correspondingnodeandclientforthei-0dfec92dserverwerenot

deletedandremainregisteredwiththeChefServer

RemovingtheChefnodeAtthispoint,theEC2instanceisbeingterminatedandremovedfromyouraccount.However,itisnotremovedfromtheChefservicethatneedstobedoneseparatelywiththenodedeletecommand.Here,theChefnodenameisspecified,nottheinstanceidentifier:

$knifenodedeletemy-first-ec2-instance

VerifythatthenodewasremovedfromChefusingnodelist:

$knifenodelist

TheoutputshouldshowyouthatyourEC2instanceisnolongerregisteredwithChef.

RackspaceCloudRackspaceCloudisanotherpopularcloud-computingproviderthatiswellsupportedbyChef.SimilartoEC2,thereisaknifepluginforRackspaceCloud:

geminstallknife-rackspace

InthesamewaythatAWSrequiresasetofcredentialstointeractwiththeAPItocreateandterminateinstances,RackspaceCloudhasitsownconfiguration.However,theRackspaceCloudAPIisalittlesimpler;youwillneedtoprovideknifewithyourRackspaceCloud’susernameandAPIkey.ForthosewhodonotalreadyhavetheirAPIkey,itcanbefoundinyourRackspaceCloudcontrolpanel.Theappropriateconfigurationtoaddtoyourknife.rbfileisasfollows:

knife[:rackspace_api_username]="YourRackspaceAPIusername"

knife[:rackspace_api_key]="YourRackspaceAPIKey"

Thisdatacanbehardcodedintoyourconfigurationfile,orsincetheknifeconfigurationfileisjustRuby,itcanbegeneratedbyevaluatingenvironmentvariablesorlookingatalocalfile.Thisisusefulifyouaresubmittingyourknife.rbfileintoasourcerepositorysothatcredentialsarenotleaked.

ProvisioninganinstanceRackspaceCloudserverprovisioningisjustasstraightforwardasitiswithEC2.Thereissomevariationinthecommand-lineoptionspassedtoknifebecauseofthewayRackspaceprovidesimagesforsystems.InsteadofusingtheinstancesizeandanAMI,youcanspecifytheflavorofthesystemtoprovision(thenode’sCPU,memory,anddiskallocation)andtheoperatingsystemtoimagetheinstancewith.Inordertodeterminewhatflavorsareavailable,theknife-rackspacepluginprovidestherackspaceflavorlistsubcommand:

$kniferackspaceflavorlist--rackspace-region=IAD

Asitispossiblethattherearedifferentcapacitiesindifferentregions,itisagoodideatocheckwhatisavailableintheregionwhereyouwanttoprovisionanode.Thiswillresultinalistofflavorsandtheirspecifications;asofnow,someofthecurrentofferingsinIADareasfollows:

IDNameVCPUsRAMDisk

2512MBStandardInstance151220GB

31GBStandardInstance1102440GB

42GBStandardInstance2204880GB

performance1-11GBPerformance1102420GB

Inadditiontoknowingwhichflavortoprovision,youneedanimageidentifier(similartoanAMI)toapplytothenewhost.Again,thislistmayvarywithregionandpossiblychangeovertimesothereisacommand,rackspaceimagelist,tolistthevarious

images:

$kniferackspaceimagelist--rackspace-region=IAD

Theoutputhereisquitelong,soithasbeensampledtoshowenoughtobeuseful:

IDName

ba293687-4af0-4ccb-99e5-097d83f72dfeArch2013.9

41e59c5f-530b-423c-86ec-13b23de49288CentOS6.5(PVHVM)

857d7d36-34f3-409f-8435-693e8797be8bDebian7(Wheezy)

896caae3-82f1-4b03-beaa-75fbdde27969Fedora18(SphericalCow)

fb624ffd-81c2-4217-8cd5-da32d32e85c4FreeBSD9.2

1705c794-5d7e-44d6-87da-596e3cf92144RedHatEnterpriseLinux6.5

df27d481-63a5-40ca-8920-3d132ed643d9Ubuntu13.10

d88188a5-1b02-4b37-8a91-7732e42348c1WindowsServer2008R2SP1

Asyoucansee,thereareanumberofLinux,BSD,andWindowsdistributionsavailabletoprovision.Inordertoprovisionanewhost,youwillusetheservercreatecommand,similartotheEC2command.Thefollowingknifecommandwillprovisiona512MBhostwithUbuntu13.10intheIADdatacenter:

$kniferackspaceservercreate-Idf27d481-63a5-40ca-8920-3d132ed643d9-f

2--rackspace-region=IAD

AssoonastheAPIrespondstotherequesttoprovisionanewhost,youwillseetheRackspacemetadataforthehost,suchastheinstanceID,name,flavor,andimage:

InstanceID:993d369f-b877-4f0f-be4b-cfc45c240654

Name:rs-21230044929009695

Flavor:512MBStandardInstance

Image:Ubuntu13.10(SaucySalamander)

Shortlyafterthis—oncethesystemhasbeenprovisioned,thenetworkinterfaceshavebeenconfigured,andtherootpasswordhasbeenset—theIPandrootpasswordwillbedisplayed:

PublicDNSName:162.209.104.248.rs-cloud.xip.io

Password:yZ3D3Tck8uGm

AfterSSHbecomesavailable,knifewillinitiatetheprocessofbootstrappingthehost.Bydefault,knifewillusethechef-fulltemplate,whichwillinstallChefviatheomnibusinstallerfortheplatformyouarebootstrapping.Thiscanbealteredbyprovidingknifewiththe–dcommand-lineoption.Assumingthatthehostisbootstrappedproperly,thesystemdatawillbedisplayedonceagainforyourinformation:

HostID:c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00

Name:rs-21230044929009695

PublicDNSName:162.209.104.248.rs-cloud.xip.io

Password:yZ3D3Tck8uGm

Environment:_default

Oncethebootstrapstepisfinished,assumingthattherearenoerrors,verifythatyournewlyprovisionedhostislistedinyourchefservice:

$knifenodelist

TheoutputwillcontainyournewlybootstrappednodeIDasspecifiedbyyouinthecommandline(via-N)orthenamegeneratedbyRackspace(inthisexample,itwillbers-21230044929009695).Congratulations!YouhaveprovisionedanewRackspaceinstancewithasinglecommand.

TerminatinganinstanceOnceyouaredonewithtesting,youmaynotwanttoleavetheEC2instancerunning,asitwillincurcostsifitremainsidle.Toensurethisdoesn’thappen,performthefollowingfoursteps:

1. ListyourRackspaceservers.2. DeletetheserverfromRackspace.3. RemovetheserverfromChef.4. VerifythattheinstancenolongerexistsinCheforRackspace.

TolistyourRackspaceinstances,usetheserverlistsubcommandoftherackspacecommand,whichwilllistalloftheRackspaceinstancesinthespecifiedregion.SimilartotheoutputfromtheEC2serverlistcommand,theoutputwilllooklikethefollowing:

$kniferackspaceserverlist--rackspace-region=IAD

InstanceIDName

993d369f-b877-4f0f-be4b-cfc45c240654rs-21230044929009695

TipSimilartotheEC2output,theresultingtableistoowideforprintsoonlytheinstanceIDandnodenameisshown.YoushouldexpecttoseepublicandprivateIPaddresses,instancetypes,andsomeotherdatathatyouwillbeabletoseeontheRackspaceCloudcontrolpanelaswell.

Youcandeleteaninstanceusingasinglecommand;theserverdeletesubcommandisinvokedwiththeRackspaceinstanceidentifiertobeterminated.Rememberthatthisisnotreversible,sothecommandwillpromptyoutoensurethatyoureallydowanttodeletetheinstance:

$kniferackspaceserverdelete993d369f-b877-4f0f-be4b-cfc45c240654--

rackspace-region=IAD

HostID:c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00

Name:rs-21230044929009695

Doyoureallywanttodeletethisserver?(Y/N)y

WARNING:Deletedserver993d369f-b877-4f0f-be4b-cfc45c240654

WARNING:Correspondingnodeandclientforthe993d369f-b877-4f0f-be4b-

cfc45c240654serverwerenotdeletedandremainregisteredwiththeChef

Server

RemovingtheChefnodeAtthispoint,theEC2instanceisbeingterminatedandremovedfromyouraccount.However,itisnotremovedfromtheChefservice;thisneedstobedoneseparatelywiththenodedeletecommand.Here,theChefnodenameisspecified,nottheinstanceidentifier:

$knifenodedeleters-21230044929009695

VerifythatthenodewasremovedfromChefwithnodelist:

$knifenodelist

TheoutputshouldshowyouthatyourrecentlycreatedRackspaceinstanceisnolongerregisteredwithChef.

SummaryTheabilitytoscaleyourinfrastructurethroughacombinationofon-andoff-sitehostsisincrediblypowerful.Ifyouneedmorecapacity,youcaneasilybringupnewhostsonEC2,RackspaceCloud,oranysimilarplatform.Additionally,thesetechniquesapplytonotonlypubliccloudprovidersbutalsotoprivatecloudplatformssuchasVMWarevSphereandothers(provided,asuitablepluginforknifeexists).

Asyouhaveseen,withChefyoucanspinupandspindowntheservercapacitytomeetyourneedswithverylittleeffort.Onceyourinfrastructuremanagementisautomated,youcanfocusonhigherlevelproblemssuchasbuildingscalableservicesandscalingtomeetyourcustomers’demands.

Expandingonthiscapability,youcouldusethesetoolstoperformthefollowing:

ManuallyincreaseordecreasethecapacityinordertomatchthedemandWriteatooltoanalyzethecurrentresourceloadandreactaccordinglyPredictthefuturecapacityandscaleappropriatelyonagivenschedule

Nowthatwehavetheabilitytobringupsomehoststoworkwith,wecantakealookathowtoworkwithcookbookstolearnhowtheyworkandhowtobuildnewones.

Chapter4.WorkingwithCookbooksCookbooksareoneofthefundamentalcomponentsoftheChefsystem.Theyarecontainersforrecipes,providers,resources,templates,andallthelogicandinformationrequiredtomanageyourinfrastructure.Thischaptercoversthefollowing:

OrganizationofcookbooksBuildingcookbooksDevelopingrecipesHandlingmultipleplatformsforacookbookorganization

CookbooksareoneofthecorecomponentsofChef.Theyare,astheirnamesuggests,acollectionofrecipesandotherdatathatwhencombinedprovideaspecificsetoffunctionalitytoasystemadministrator.Ineachcookbook,youwillfindacollectionofdirectoriesandfilesthatdescribethecookbookanditsrecipesandfunctionality.Thecorecomponentsofacookbookareasfollows:

CookbookmetadataAttributesRecipesTemplatesDefinitionsResourcesProvidersRubylibrariesSupportfiles

Acookbookisacollectionoffilesanddirectorieswithawell-knownstructure.Noteverycookbookhasallofthesecomponents.Forexample,theremaybenoneedtodevelopcustomresourcesorprovidersinacookbookthatonlyusesChef-suppliedresources.However,everycookbookdoesneedtohaveametadatafilethatprovidesvariousbitsofinformationsuchasitsname,version,dependencies,andsupportedsystems.

Let’stakealookatthememcachedcookbookasitisareasonablysimplecookbookthatiscapableofinstallingandconfiguringthememcacheserver,whichisadistributedmemory-backedcacheservice:

Here,youcanseethatthiscookbookcontainsattributes,definitions,recipes,andtemplates,aswellasafilenamedmetadata.rb(themetadatafile)andaREADME.mdfile.Itisagoodideatoprovideexamplesofhowtouseyourcookbookandrecipesinsomesortofdocumentation,suchasaREADME.mdfile.Whenyoulookattheprecedingscreenshot,youwillseethatthedirectorynamesmaptothecomponentnamesandeachcontainssomefilesorsubdirectorieswithfiles.WewilldiscusstheorganizationofthespecificChefcomponentsingreaterdetailfurtheronaswediveintomoredetailsoneachtypelater.Fornow,itissufficienttoknowthatthedirectorystructureisdesignedtogrouptogetherfilesforeachtypeofcomponent.Alsonoticethat,asmentionedearlier,thiscookbookisanexampleofonethatdoesnothaveallthecomponents,astherearenonewresourcesorprovidersinthiscookbook.

SomeofthesefilesarepurelyinformationalandhavenoeffectonyourrecipesorChefitself,suchastheREADME.mdfile.Thisfile,andotherssuchasCHANGELOG,LICENSE,orDEVELOPMENTfiles,isincludedtoconveyinformationtoyouabouthowtoparticipate,license,orotherwiseusethecookbook.

Thereisalotofinformationthatisstoredinsideacookbook—thisinformationincludesthestepstotakeinordertoachieveadesiredeffectsuchastheinstallationofaserviceorprovisioningofusersonahost.Ahigh-leveloverviewofthecontentthatwewillbelearningaboutinthischapter,sothatyouhaveanideaofhowthecomponentsworktogetherbeforeyoulearnaboutthemindepth,isshownasfollows:

Attributes:Theseareattributesthatthecookbook’srecipesrelyon.Awell-definedcookbookshouldcontainsomesanedefaultsfortherecipessuchasinstallationdirectories,usernames,downloadableURLs,andversionnumbers.Anythingarecipeexpectsthenodetohavedefinedshouldbegivenadefaultvaluesothattherecipewillbehaveasexpected.Recipes:Rubyscriptsdefinetherecipesinthecookbook.Acookbookcancontainasfewasoneorasmanyrecipesasitsauthorwouldliketoputintoit.Mostpackage-specificcookbooksonlycontainafewrecipes,whilesomecookbooks,suchasyour

organization’sprivatecookbook,mayhavedozensofrecipesforinternaluse.Templates:TheseareRubyERBfilesthatareusedtodescribeanyfilethatneedstohavesomedynamicdatainit;often,theseareusedforstartupscriptsorconfigurationfiles.Resources:Thesedescribearesourcethatcanbeusedinarecipe.ResourcesareRubyscriptsthatuseChef’sresourcedomain-specificlanguage(DSL)todescribevariousactions,attributes,andotherpropertiesoftheresource.Providers:Thesedescribeanimplementationofaresource;inthecaseofthesupervisordcookbook,theserviceproviderfileoutlinestheactualimplementation-specificlogicoftheactionsthataresourcecanperform.Therearemanytypesofservicesthatyoucouldhave:supervisord,runit,monit,bluepill,andsoon.

Additionally,cookbooksmayincludeavarietyofsupportfilesthatarenotdirectlypartoftherecipes,suchasthefollowing:

Definitions:Theseareusedtobuildreusabletemplatesforresources.Perhapsyouwanttodefinethestructureofauseraccount,abackgroundworker,orarunnableprocess.Theseareawaytoprogrammaticallydescribewhattheselooklikeandimplementanylogictheymightneed.Rubylibraries:Anyreusablecodethatyourrecipesneedcanbeincludedinthecookbook.Thingsthatgoinhereareaccessiblebyyourrecipesandautomaticallyloadedforyou.SupportFiles:Thesearearbitrarydatafilesthatdon’tfallintoanyoftheothercategories.Tests:Recipes,composedofRubycode,canincludeunittestsorcucumberteststoverifythatthelogicworks.Notethatthesetestsareunittests,notintegrationtests;theyarenotdesignedtoensurethatyouhaveconfiguredyournodesproperlyorthattherearenoconflictsorotherissueswhenapplyingtheserecipes.

AttributesAttributesareChef’swayofstoringconfigurationdataandcanbethoughtofasalarge,butdisjointed,hashstructure.Chefpullsdatafromvariouslocationsandcombinesthatdatainaspecificordertoproducethefinalhashofattributes.Thisdataiscomputedwhenaclientrequestsitsrunlistfromtheserver(suchaswhenyouexecutechef-clientonanode).Thismechanismallowsyoutodescribedatawithahigherlevelofspecificityateachstepoftheprocess,decreasinginscopegoingfromthecookbookattributesfilesdowntonode-specificconfigurationdata.

Forexample,imagineyouaredeployingPostgreSQLontothehostsinyourinfrastructure.WithPostgreSQL,thereareaverylargenumberofconfigurationoptionsthatcanbetuned,rangingfromopenportsandnumberofconcurrentconnectionsdowntomemoryusedforkeycachesandotherfine-grainedconfigurationoptions.Thecookbook’sattributesfilesshouldprovideenoughconfigurationsforPostgreSQLtoworkwithoutmakinganymodificationtoahostandwithoutotherthingsbeingdeployed;also,theywouldmostlikelycontainaprettyvanillasetofconfigurationvalues,whichatahighlevelmightlooklikethefollowing:

InstallVersion9.3fromthesourcecodeListenonPort5432onIP0.0.0.0/0Storedatain/var/lib/postgresqlCreateanduseapgsqluser

Attributedatahasnotonlyanumberofsourcesthatitcanbepulledfrombutalsoasetofpriorities:default,normal,andoverride(inincreasingorder).Withineachlevel,dataispulledfromthecookbookattributesfilesandthenfromtheenvironment,role,andnodeconfigurationdatastoredintheChefserver(inthatorder).Combined,thisprovidesacomprehensivemechanismtodefineandcustomizethebehaviorofyourrecipesastheyareappliedtothenodes.

Now,asyoucanimagine,thisisfineforanumberofinstallationswheretheserverhasallofthespaceallocatedontherootmountpoint,ordoesn’thavesecurityrestrictionsaboutwhichIPaddressesshouldbelistenedon,andsoon.Itwouldbenicetobeabletosaythatinproduction,wewanttouseVersion9.3,butinatestenvironment,wewanttoinstallVersion9.4inordertoperformsometeststhatwedon’twanttoruninproduction.Wemayalsowanttospecifythatinproduction,ourhostsareEC2instanceswithacustomizedEBSRAIDforourPostgreSQLdataandsothedatashouldbestoredin/vol/ebs00/postgresql.Usingthismultilayeredapproachforconfigurationdata,thisisentirelypossible.

AttributefilescontainRubycodethatstorestheconfigurationdata.Inthiscase,toachieveourdescribeddefaultbehavior,wecouldhaveafile,attributes/default.rb,thatcontainsthefollowingtext:

default['postgresql']['listen_address']="*"

default['postgresql']['data_dir']="/var/lib/postgresql"

default['postgresql']['install_method']="source"

default['postgresql']['version']="9.3"

ThehashthatthisdescribeswilllooklikethefollowingJSONdictionary:

'node':{

'postgresql':{

'port':'5432',

'listen_address':'*',

'data_dir':'/var/lib/postgresql',

'install_method':'source',

'version':'9.3'

Now,asdescribed,wewanttooverridetheversioninourstagingenvironmenttoinstallVersion9.4;thismeansthatinourstagingenvironment,theconfiguration(exactlyhowtomakethischangewillbediscussedlater)willneedtohavethefollowinginformation:

'node':{

'postgresql':{

'version':'9.4',

WhentheChefclientrunsonanodeinthestagingenvironment,theChefserverknowsthatthenodeisinthestagingenvironmentandwilltakethestageconfigurationaboveandoverlayitontopofthedefaultsspecifiedinthecookbook.Asaresult,thefinalconfigurationdictionarywilllooklikethefollowing:

'node':{

'postgresql':{

'port':'5432',

'version':'9.4'

Noticethattheversionhasbeenchanged,buteverythingelseremainsthesame.Inthisway,wecanbuildveryspecificconfigurationsforourhoststhatpullinformationfromavarietyofplaces.

ItisimportanttonotethatbecausetheseareinterpretedRubyscripts,theircontentscanrangefromsimpleattribute-settingstatementstocomplexlogicusedtodetermineanappropriatesetofdefaultattributes.However,it’sworthrememberingthatthemorecomplicatedyourconfigurationis,theharderitmaybetounderstand.

MultipleattributefilesChefloadsattributefilesinalphabeticalorderandcookbookstypicallycontainonlyoneattributefilenameddefault.rb.Insomecases,itmakessensetoseparatesomeoftheattributesintoseparatefiles,particularlywhentherearealotofthem.Asanexample,thecommunity-maintainedMySQLcookbookhastwoattributefiles:server.rbfortheserverattributesandclient.rbwithclient-specificattributes.Eachfilecontainsanywherebetween50and150linesofRubycode,soitmakessensetokeepthemseparateandfocused.

SupportingmultipleplatformsTherearetimeswhenasimpleattributesfiledoesn’tmakesense,sobeingabletodynamicallydefinethedefaultsisveryuseful.Consideramultiplatformcookbookthatneedstoknowwhichgrouptherootuserisin.Thenameofthegroupwillvaryaccordingtotheoperatingsystemoftheendhost.IfyouareprovisioningaFreeBSDorOpenBSDhost,thenthegroupfortherootwillbewheel,butonanUbuntumachine,thegroupisnamedadmin.TheattributesfilecanuseplainoldRubycodeoroptionalChef-providedconveniencemethodssuchasvalue_for_platform,whichisaglorifiedbutcompactswitchstatement:

default[:users]['root'][:group]=value_for_platform(

"openbsd"=>{"default"=>"wheel"},

"freebsd"=>{"default"=>"wheel"},

"ubuntu"=>{"default"=>"admin"},

"default"=>"root"

LoadingexternalattributesSometimesitisusefultoloadattributesfromanothercookbook;ifyourcookbookistightlycoupledtoanothercookbookorprovidessomeextendedfunctionality,itmaymakesensetousethem.Thiscanbeachievedintheattributesfilebyusingtheinclude_attributemethod(again,thisisaChef-specificconveniencemethod).

Let’sconsiderthatyouwanttoknowtheportthatApacheisconfiguredtouse.Youcouldusetheportkeyfromtheapacheconfigurationsection,butitisnotguaranteedthatitexists(itmaynothavebeendefinedortherecipethatcontainsitmaynothavebeenloadedyet).Toaddressthis,thefollowingcodewouldloadthesettingsfromattributes/default.rbinsideoftheapachecookbook:

include_attribute"apache"

default['mywebapp']['port']=node['apache']['port']

Ifyouneedtoloadanattributesfileotherthandefault.rb,sayclient.rb,insidethepostgresqlcookbook,youcanspecifyitinthefollowingmanner:

include_attribute"postgresql::client"

Makesurethatanycookbooksyourelyonarelistedasadependencyinyourcookbook’smetadata.Withoutthis,theChefserverwillhavenowayofknowingthatyourrecipesor

configurationdatadependonthatcookbook,andsoyourconfigurationmayfailasaresultofthis.

UsingattributesOnceyouhavedefinedyourattributes,theyareaccessibleinourrecipesusingthenodehash.Chefwillcomputetheattributesintheorderdiscussedandproduceonelargehash,whichyouwillhaveaccessto.

TipChefusesaspecialtypeofhash,calledaMash.Mashesarehasheswithwhatisknownasindifferentaccess—stringkeysandsymbolkeysaretreatedasthesame,sonode[:key]isthesameasnode["key"]).

IfwehadthePostgreSQLattributesanduserattributesasspecifiedpreviously,withoutanyoverrides,thentheresultingconfigurationwilllooklikethefollowing:

'node':{

'postgresql':{

'port':'5432',

'version':'9.3'

'users':{

'root':{'group'=>'wheel'},

Thisdatacouldthenbeaccessedanywhereinourrecipesortemplatesthroughvariablessuchasnode[:postgresql][port]ornode[:users][:root][:group].Rememberthatthefinalversionofthenode’sconfigurationdataisdeterminedatthetimetheclientmakestherequestforitsconfiguration.ThismeansthatChefgeneratesasnapshotofthecurrentstateofthesystem,collapsedaccordingtoitsrulesofprecedence,forthatnodeandpassesittothehosttoperformitsoperations.

MetadataEachcookbookcontainsametadata.rbfileintherootdirectoryofthecookbookthatcontainsinformationaboutthecookbookitself,suchaswhomaintainsit,thelicense,version,containedrecipes,supportedplatforms,andthecookbook’sdependencies.ThecontentsofthisscriptareusedtogenerateaJSONfilethatdescribesthecookbook,whichisusedbytheChefserverfordependencyresolution,searchingandimportingintorunlists.

Thisisarequiredfileforacookbook,andhereisanexamplemetadata.rbfilefromthePostgreSQLdatabaseserver,whichisslightlymodifiedtofitthefollowing:

name"postgresql"

maintainer"Opscode,Inc."

maintainer_email"cookbooks@opscode.com"

license"Apache2.0"

description"InstallsandconfiguresPostgreSQL"

long_descriptionIO.read(File.join(

File.dirname(__FILE__),'README.md'

version"3.3.4"

recipe"postgresql","Includespostgresql::client"

recipe"postgresql::ruby","InstallsRubybindings"

recipe"postgresql::client",

"Installsclientpackage(s)"

recipe"postgresql::server","Installsserverpackages"

recipe"postgresql::server_redhat",

"InstallsRedHatserverpackages"

recipe"postgresql::server_debian",

"InstallsDebianserverpackages"

%w{ubuntudebianfedorasuseamazon}.eachdo|os|

supportsos

%w{redhatcentosscientificoracle}.eachdo|el|

supportsel,">=6.0"

depends"apt"

depends"build-essential"

depends"openssl"

Becausethemetadata.rbfileisaRubyscript,itallowsyoutousearbitraryRubycodeinsideofit.Here,forexample,thelong_descriptionentryisgeneratedprogrammaticallybyreadinginthecontentsofthesuppliedREADME.mdfile:

long_descriptionIO.read(File.join(File.dirname(__FILE__),

'README.md'))

Here,thePostgreSQLcookbooksupportsmultipleplatforms,soinsteadofwritingeachplatformthatissupportedonalineofitsown,youcouldusealoopsimilartotheoneusedinthemetadata.rbfile:

%w{ubuntudebianfedorasuseamazon}.eachdo|os|

supportsos

Additionally,ifitonlysupportscertainplatformswithaminimumversion,youcouldwritesomethingsimilartothefollowing,whichdeclaressupportforRedHat-baseddistributionsgreaterthan(orequalto)Version6.0:

%w{redhatcentosscientificoracle}.eachdo|el|

supportsel,">=6.0"

Inthiscookbook,thedependenciesarelistedlinebylinebutcouldberepresentedsimilarlyifyouhavealargenumber:

depends"apt"

depends"build-essential"

depends"openssl"

Dependenciescouldalsoberewrittenasfollows:

%w{aptbuild-essentialopenssl}.eachdo|dep|

dependsdep

Obviously,inthiscase,youaren’tsavinganyroom;however,ifyouhadtenormoredependencies,itcouldmakeitmorecompact.

AslongasyourRubycodeproducessomethingthatisacompatibleargumentorconfiguration,youcanbeascleverasyouwant.Takeadvantageofyourabilitytodynamicallygenerateaconfiguration.

RecipesRecipesarethecorecomponentofgettingthingsdone.Theyarescripts,writteninRuby,thatprovidetheinstructionstobeexecutedonendhostswhentheChefclientisrun.Recipesareplacedintherecipesdirectoryinsideofacookbook,andeachrecipeisdesignedtoachieveaspecificpurpose,suchasprovisioningaccounts,installingandconfiguringadatabaseserver,andcustomsoftwaredeployments.

Recipescombineconfigurationdatawiththecurrentstateofthehosttoexecutecommandsthatwillcausethesystemtoenteranewstate.Forexample,aPostgreSQLdatabaseserverrecipewouldhavethegoalofinstallingandstartingaPostgreSQLserveronanyhostthatrunstherecipe.Let’slookatafewpossiblestartingstatesandtheexpectedbehavior:

AhostwithoutPostgreSQLinstalledwouldbeginatthestateofnothavingtheservice;then,itwillexecutethecommandsrequiredtoinstallandconfiguretheserviceHostswithanexistingbutoutdatedPostgreSQLservicewouldbeupgradedtothelatestversionofthedatabaseserverHostswithacurrentinstallationofPostgreSQLwouldhaveitsPostgreSQLinstallationuntouchedInallcases,theconfigurationonthediskwouldbeupdatedtomatchtheconfigurationstoredintheChefserver

Toachievethesegoals,recipesuseacombinationofresources,configurationdata,andRubycodetodeterminewhattoexecuteontheendhost.Eachhost-levelresource—files,configurationfiles,packages,users,andsoon—ismappedtoaChefresourceinarecipe.Forexample,considertherecipethatwesawearlierinthebookthatwasusedtodemonstratethattheChef-soloinstallationwasfunctional:

action:create

Thisisacompleterecipe;ithasonesteptocreateafile,andthatisallitdoes.Thefilebeingcreatedontheendhostneedsaname;here,itwillbenamedENV['HOME']/example.txt,whichisRuby’swayofrepresenting$HOME/example.txt.Inadditiontoaname,weareinstructingCheftocreatethefile(wecouldjustaseasilyinstructCheftodeletethefile)andtoputthecontentsGreetings$USERintothefile,replacingwhatisinthere.

Wecouldextendourrecipetoensurethataspecificuserexistedonthehostandcreateafilewiththeownersettothenewlycreateduser:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

file"/tmp/agent.txt"do

action:create

content"Hellofromsmith!"

owner"smith"

group"agents"

mode"0660"

Eachrecipeisascriptthatisrunfrombeginningtoend,assumingthatnothingcausesittoabort.Also,eachrecipecanaccessthenode’sattributedataandleverageresourcestocompiletemplates,createdirectories,installpackages,executecommands,downloadfiles,anddojustaboutanythingthatyoucandofromashellasanadministrator.Ifyoucan’taccomplishwhatyouneedtodousingtheexistingChefresources,youcaneithercreateyourowncustomresources,oryoucanexecuteanarbitraryuser-definedshellscript.

ResourcesResourcesareprogrammaticbuildingblocksinChef;theyareadeclarativemechanismtomanipulatearesourceonahost.Resourcesdeliberatelyhidetheunderlyingimplementationthatislefttoaprovider.Itisimportanttorecognizethataresourcedescribeswhatisbeingmanipulated,nothowitisbeingmanipulated;thisisbydesign,asitprovidesahighlevelofabstractionforChefrecipestobeasplatform-neutralaspossible.

Forexample,Chefhasbuilt-inresourcesthatincludethefollowing:

CronjobsDeploymentsFilesystemcomponents(mountpoints,filesanddirectories,andsoon)Sourcecoderepositories(Gitandsvn)LogsPowerShellscripts(Windowstargets)ShellscriptsTemplatesPackagesUsersandgroups

Resourcescombinedwithproviders(discussedshortly)arecollectivelyreferredtoasLWRPsorlightweightresourceproviders;theymakeupalargeportionofthefunctionalitywithinaChefrecipe.

Resourcesarecomposedofaresourcename(packagename,filepath,servicename,andsoon),anaction,andsomeattributesthatdescribethatresource.

UsingresourcesResources,aswehaveseeninsomeexamples,takethefollowingform:

resource_name<nameparameter><ruby_block>

Intheprecedingcode,resource_nameistheregisterednameoftheresource,suchasfile,package,anduser.Thenameparameterisaspecialargumenttotheresourcethatgivesthisresourceauniquename.Thisisoftenalsousedbytheresourceasadefaultvalueforanattributethatnaturallymapstothenameoftheresourcesuchasfilename,username,andpackagename(youcanseeapatternhere);however,youcanuseanarbitrarynameattributeandmanuallysettheattribute.TheRubyblockisjustablockofcodeinRuby;thisishowChef’sDSLworks.InChef,eachresourceexpectssomespecificthingsinsideitsconfigurationblock.Youwillfindthatmanyresourceshavedifferentexpectations,butingeneral,aresourceblockinarecipewillbeofthefollowingform:

resource_name"nameattribute"do

attribute"value"

Thepreviousexample,whichcreatedanewuser,wasthefollowing:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

Here,theresourcenameisuser,thenameattributeissmith,andtheblockofcodebeingpassedtotheresourcehassevenattributes:action,system,comment,uid,gid,home,andshell.Eachoftheseattributeshasavalueassociatedwithit;internally,theRubycodefortheresourcewillstoretheseinsomevariablestobeusedwhenmanipulatingthespecifiedresource.Inthiscase,constructingauserontheendhostthroughthecorrectproviderwillbehelpful.

Oneoftheseattributes,action,isabitunique;itsjobistotelltheresourcewhatactiontotakeontheresource.Thelistofavailableactionswillbedifferentwitheachresource,buttypically,aresourcewillhaveactionssuchascreate,delete,orupdate.Havealookatthedocumentationfortheresourceyouareworkingwith;thedocumentationwilldescribetheavailableactionsandwhattheydoseparatelyfromtheotherattributes.

Todemonstratehowthenameattributeisusedasadefaultvaluefortheuserresource,thefollowingrecipehasthesamebehaviorasthepreviousone,buthasoneminorchange:

user"agent_smith"do

username"smith"

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

Here,youcanseethatanadditionalattribute,username,hasbeenaddedtotheresourceblockwiththevaluethatwaspreviouslyinthenameattribute.Additionally,thenameattributehasbeenchangedto"agent_smith".Ifyouweretoexecutethisrecipeorthepreviousexample,itwouldhavethesameeffect:tocreatealocalsystemuser,smith,withtheUID,GID,home,andotherattributesspecified.

OverridingadefaultbehaviorInadditiontoproperties,resourcesalsohaveadefaultaction.Moreoftenthannot,thedefaultactioniscreate,butagain,youwillwanttoconsultthedocumentationfortheresourceyouareworkingwithtomakesurethatyouknowwhatthedefaultbehaviorisforaresource.Youdon’twanttoaccidentallydestroysomethingyouthoughtyouwerecreating!

Aconcreteexamplemightbeinstallingthetcpdumppackageonyoursystem.Toinstallthedefaultversionwithnocustomization,youcouldusearesourcedescriptionsuchasthefollowing:

package"tcpdump"

Thisworksbecausethedefaultactionofthepackageresourceistoperformaninstallation.Ifyoulookatthesourcecodeforthepackageresource,youwillseethefollowingatthebeginningoftheconstructor:

definitialize(name,run_context=nil)

@action=:install

@allowed_actions.push(:install,:upgrade,

:remove,:purge,:reconfig)

@candidate_version=nil

@options=nil

@package_name=name

Thistellsusthatthedefaultaction,ifunspecified,istoinstallthepackageandtousethenameattributeasthepackagename.So,theprevioussimpleone-lineresourceisthesameaswritingoutthefollowingblock:

package"tcpdump"do

action:install

package_name"tcpdump"

Here,however,package_namewilldefaulttothenameattribute,sowedonotneedtoprovideitiftheresourcenameisthesameasthepackageyouwishtoinstall.Additionally,ifyouwantedtobemoreverbosewithyourresourcedescriptionandinstallaspecificversionofthetcpdumppackage,youcouldrewritethepackageresourcetolooksomethinglikethefollowing:

package"tcpdump"do

action:install

version"X.Y.Z"

Ifyoureadthedocumentationforthepackageresourceorexaminethefullconstructorforthepackageclass,youwillseethatithasanumberofotherattributesaswellaswhattheydoandwheretheirdefaultvaluescomefrom.Alltheresourcesfollowthisform;theyaresimplyRubyclassesthathaveanexpectedstructure,whichtheyinheritfromthebaseresourceclass.

TemplatesChefdynamicallyconfiguressoftwareandhosts,andalargepartofconfiguringUNIX-basedsystemsandsoftwareinvolvesconfigurationfiles.Chefprovidesastraightforwardmechanismtogenerateconfigurationfilesthatmakeiteasytocombineconfigurationdatawithtemplatefilestoproducethefinalconfigurationfilesonhosts.ThesetemplatesarestoredinthetemplatesdirectoryinsideofacookbookandusetheERBtemplatelanguage,whichisapopularandeasy-to-useRuby-basedtemplatelanguage.

Whyusetemplates?Withouttemplates,yourmechanismtogenerateconfigurationfileswouldprobablylooksomethinglikethis:

File.open(local_filename,'w')do|f|

f.write("<VirtualHost*:#{node['app]['port']}")

f.write("</VirtualHost>")

Thisshouldbeavoidedforanumberofreasons.First,writingconfigurationdatathiswaywouldmostlikelymakeyourrecipeveryclutteredandlengthy.Secondly,andmoreimportantly,itviolatesChef’sdeclarativenature.Bydesign,Chefprovidesyouwiththetoolstodescribewhattherecipeisdoingandnothowitisdoingit,whichmakesreadingandwritingrecipesmucheasier.Simplerrecipesmakeforsimplerconfiguration,andsimplerconfigurationscalesbetterbecauseitiseasiertocomprehend.Handrollingaconfigurationfileistheoppositeapproach;itveryspecificallydictateshowtogeneratethefiledata.Considerthefollowingcode:

config_file="#{node['postgresql']['dir']}/postgresql.conf"

pgconfig=node[:postgresql]

File.open(config_file'w')do|f|

f.write("port=#{pgconfig[:port]}")

f.write("data_dir=#{pgconfig[:data_dir]}")

f.write("listen_address=#{pgconfig[:listen_address]}")

File.chown(100,100,config_file)

File.chmod(0600,config_file)

ThiscodegeneratesaPostgreSQLconfigurationfilefromtheattributehash,onelineatatime.Thisisnotonlytime-consumingandhardtoreadbutpotentiallyveryerror-prone.Youcanimagine,evenifyouhavenotpreviouslyconfiguredanyPostgreSQLservers,justhowmanyf.write(...)statementscouldbeinvolvedingeneratingafullpostgresql.conffilebyhand.Contrastthatwiththefollowingblockthatleveragesthebuilt-intemplateresource:

template"#{node['postgresql']['dir']}/postgresql.conf"do

source"postgresql.conf.erb"

owner"postgres"

group"postgres"

mode0600

Theprecedingblockcouldbecombinedwithatemplatefilethatcontainsthefollowingcontent:

<%node['postgresql']sort.eachdo|key,value|%>

<%nextifvalue.nil?-%>

<%=key%>=<%=

casevalue

whenString

"'#{value}'"

whenTrueClass

whenFalseClass

<%end%>

Ifwetakeourtemplateandthenapplythefollowingattributedataaswehadshownpreviously,thenwewouldhavegeneratedtheexactsameconfigurationfile:

'node':{

'postgresql':{

'port':'5432',

'version':'9.3'

'users':{

'root':{'group'=>'wheel'},

Onlynowwecanuseatemplatethatishighlyflexible.Ourtemplateusesthekey-valuecombinationsstoredintheconfigurationhashtodynamicallygeneratethepostgresql.conffilewithoutbeingchangedeverytimeanewconfigurationoptionisaddedtoPostgreSQL.

ChefusesERB,atemplatelanguage,thatisprovidedbythecoreRubylibrary.ERBiswidelyavailableandrequiresnoextradependencies;itsupportsRubyinsideoftemplatesaswellassomeERB-specifictemplatemarkup.

AquickERBprimerAsERBisverywelldocumentedandwidelyused,thisportionofthechapterservesonlyasaquickreferencetosomeofthemostcommonlyusedERBmechanisms.Formoreinformation,seetheofficialRubydocumentationathttp://ruby-doc.org/stdlib-2.1.1/libdoc/erb/rdoc/ERB.html.

ExecutingRuby

ToexecutesomearbitraryRubycode,youcanusethe<%%>container.The<%partindicatesthebeginningoftheRubycode,and%>indicatestheendoftheblock.Theblockcanspanmultiplelinesorjustonesingleline.Examplesofthisareasfollows:

ERBcode Output

[1,2,3].eachdo|index|

putsindex

<%users.collect{|u|

putsu.full_name}%>

BobSmith

SallyFlamingo

YoucanmixRubyandnon-Rubycode(usefultorepeatblocksofnon-Rubytext)asfollows:

<%[1,2,3].eachdo|value|%>

Non-rubytext…

<%end%>

Thiswouldyieldthefollowing:

Non-rubytext…

Variablereplacement

ERBhasasyntaxtoreplaceasectionofthetemplatewiththeresultsofsomeRubycoderatherthanrelyingonprintstatementsinsideyourRuby.Thatcontainerissimilartothelastone,withtheadditionoftheequalsigninsidetheopeningtag.Itlookslike<%=%>.AnyvalidRubycodeisacceptableinsidethisblock,andtheresultofthiscodeisputintothetemplateinplaceoftheblock.Examplesofthisareasfollows:

<%=@somevariable%>

<%=hash[:key]+otherhash[:other_key]%>

<%=array.join(",")%>

Thiscanbecombinedwiththepreviousexampletoproducecomplexoutput:

<%[1,2,3].eachdo|value|%>

Thevaluecurrentlyis<%=value%>

<%end%>

Thiswouldyieldthefollowing:

Thevaluecurrentlyis1

UsingjustthesebasicfeaturesofERB,combinedwithChef’sconfigurationdata,youcanmodeljustaboutanyconfigurationfileyoucanimagine.

ThetemplateresourceChefprovidesatemplateresourcetogeneratefilesviatemplates.Therearethreekeyattributesofthetemplateresource,whichareasfollows:

path:Thisspecifieswheretoputthegeneratedfilesource:Thistellstheresourcewhichtemplatefiletousevariables:Thisspecifieswhatdatatopopulatethetemplatewith

Thepathattributeusesthenameattributeasitsdefaultvalueandpopulatesthetemplatespecifiedbyasourcefilewiththedatapassedtoitthroughthevariablesattribute.Templatesarecontainedinsideofthetemplatesdirectory,whichisplacedinsideofacookbook;ifasourceisnotspecified,itwillbeexpectedthatafileexistsinsidethedirectorywiththesamenameasthepath,whichisonlyrootedinthetemplatesdirectorywitha.erbextension.Hereisasimpletemplateresourceexample:

template"/etc/motd"do

variables:users=>["Bart","Homer","Marge"]

Thisresourcewillexpectthatafileexistsinthetemplate’ssearchpath(moreonhowthatisdeterminedinabit)asetc/motd.erb,anditthenexposesanarrayofthreestringsasausersvariableandwritestheresultsoutas/etc/motdonthehost.ThecorrespondingMOTDtemplatecouldlooklikethefollowing:

Welcometocrabapple.mydomain.com!Ournewestusersare:

<%@users.eachdo|user|%>

*<%=user%>

<%end%>

ThetemplatevariablesTherearetwoprimarysourcesofdataforatemplate:datapassedexplicitlythroughtheresourceblockattributesandnodeconfigurationdata.Explicitvariablesareuserdefinedintherecipeandmaybeusedtooverridesomesettingsorpassinconfigurationthatisdynamicallygeneratedinsidetherecipe.ThenodeconfigurationdataiscomputedbyChefatruntimeandrepresentsasnapshotofthecurrentconfigurationthatwillbeappliedtothenode.

PassingvariablestoatemplateSometimesyouwillneedtopassdatatoatemplatefrominsideyourrecipeinsteadofrelyingontheglobalnodeattributes.Perhapsyouhavesomelogicthatcomputessomevariabledata,butitdoesn’tbelonginthenodehash;Chefsupportsdoingjustthisinthetemplateresource.ThedatapassedexplicitlyisavailabletotheERBtemplateasaninstancevariable,prefixedinRubywiththe@symbol.Forexample,considerthefollowingrecipesnippet:

config_hash={

:food=>"asparagus",

:color=>"blue"

template"/etc/myapp.conf"do

source"myapp.conf.erb"

owner"root"

mode"0664"

variables(

:install_path=>"/opt/#{hostname}/myapp",

:config=>config_hash

The:install_pathand:configkeysareaccessibleinthetemplateasinstancevariableswiththesamename.Theywillbeprefixedbythe@characterandcouldbeusedinatemplatesimilartothefollowing:

database_path="<%=@install_path%>/db"

storage_path="<%=@install_path%>/storage"

<%config.eachdo|key,value|%>

<%=key%>="<%=value%>"

<%end%>

Here,thetemplateexpectsaspecifickey,install_path,todeterminewheretostorethedatabase;thekey-valuehashspecifiedbyconfigisthenusedtogeneratesomearbitraryconfigurationsettingsinthetemplate.

AccessingcomputedconfigurationsInadditiontodatapassedviathevariablesattribute,atemplatecanaccessanode’scomputedconfigurationdatathroughthenodelocalvariable.ThisisaccessedasaRuby

hash,whichwillbestructuredsimilarlytoadictionaryorahashinanyotherlanguage.InourpreviousPostgreSQLattribute’sdataexample,thefollowingvaluesweredefined:

default['postgresql']['listen_address']="*"

Evenifnootherconfigurationdatasupersedestheseconfigurationvalues,therewillbeapostgresqlkeyinthenode’sconfigurationdatathatcontainsthekey’sportandlisten_address.Usingthisinformation,wecanwritearecipethatusesatemplateresourceandamatchingtemplatelikethefollowing:

template"/etc/postgresql/postgresql.conf"do

source"postgresql.conf.erb"

owner"psql"

mode"0600"

listen_addresses='<%=node[:postgresql][:listen_address]'

port=<%=node[:postgresql][:port]%>

Whenthedefaultattributesdataiscombinedwiththeexampletemplate,theresulting/etc/postgresql/postgresql.conffilewillhavethefollowingcontent:

listen_addresses='*'

post=5432

Aspreviouslydiscussed,thecomputedattributeshashforagivennodecomesfromavarietyofsources.Thosesourcesincludeattributesfilesinthecookbook,role,environment,andnode-levelconfigurationvaluesstoredinChef,eachwithitsownlevelofprecedence.

SearchingfortemplatesAsyouhavelikelynoticed,Chefattemptstoallowyouasmuch,oraslittle,specificityasyouwantwhendefiningyourconfiguration,andtemplatesarenodifferent.Justasthefinalnodeconfigurationiscomputedfromavarietyoflocations,thetemplatesdirectoryhasaspecificsearchorder.Thisallowstheauthorofthecookbooktoprovideasetofdefaulttemplatesaswellassupportplatformandhost-specificoverrides.

Thedefaulttemplatedirectoryshouldbeusedtoprovidedefaultversionsofthetemplates.Anyplatform-orhost-specificdirectoriesareplacedalongsideitandwillbeusedwhenappropriate.Thesearchorderforatemplateisasfollows:

HostnameDistributionversionDistributionDefaultlocation

Asanexample,let’sconsiderascenarioinwhichweappliedarecipewiththepostgresql.conf.erbtemplateresourcetoanode,db1.production.mycorp.com,whichisrunningDebian6.0.Chefwillthenlookforthefollowingfilesinsideofthetemplatesdirectory:

host-db1.production.mycorp.com/postgresql.conf.erb

debian-6.0/postgresql.conf.erb

debian/postgresql.conf.erb

default/postgresql.conf.erb

Thesearchisperformedinthatorderwiththefirstmatchbeingthechosentemplate,applyingthehighestlevelofspecificitybeforethelowest(asisthepatternwithotherChefmechanisms,includingconfigurationdata).

Thisdifferentiationofconfigurationfilesbyhost,platform,andevenversionisveryuseful.Itallowsyoutoprovideasanesetofdefaultswhilesupportinghost-orsystem-specificquirkssimultaneously.

DefinitionsSometimes,youfindthatyouarecreatingsomethingrepeatedlyand,similartoaconfigurationtemplate,youneedatemplatetogenerateobjectsofagiventype.SomeexamplesofthismightbeApachevirtualhosts,aspecifictypeofapplication,oranythingelsethatisrepeatedalot.Thisiswheredefinitionscomein,andtheyarestoredinthedefinitionsdirectoryinsideofacookbook.

Definitionsareloadedandavailableasnamedresourcesjustasotherresourcessuchaspackages,files,andsoonare;theonlydifferenceisthatthereisnoprovider.Youcanthinkofthemasresourcesandprovidersallinone.Subsequently,theyaremuchmorerigidandlimitedinscopethananormalresourcewouldbe.HereisanexampledefinitiontoinstallPythonlibrariesusingpipandarequirements.txtfile:

define:pip_requirements,:action=>:rundo

name=params[:name]

requirements_file=params[:requirements_file]

pip=params[:pip]

user=params[:user]

group=params[:group]

ifparams[:action]==:run

script"pip_install_#{name}"do

interpreter"bash"

user"#{user}"

group"#{group}"

code<<-EOH

#{pip}install-r#{requirements_file}

only_if{File.exists?("#{requirements_file}")andFile.exists?("#

{pip}")}

Here,wearedeclaringanewtypeofdefinition,apip_requirementsobject.Thislooksandbehavessimilarlytoaresource,exceptthatitismuchsimpler(andlessflexible)thanaresource.Ithassomeattributes,whichareloadedviathespecialparamsargument,andcontainsalittlebitoflogicwrappedaroundascriptresource.Let’stakealookathowitwouldbeusedandthenseehowitworks:

pip_requirements"my_requirements"do

pip"#{virtualenv}/bin/pip"

usernode[:app][:user]

groupnode[:app][:group]

requirements_file"#{node[:app][:src_dir]}/requirements.txt"

Hereyouseewhatlookslikearesource,butisinfactadefinition.Asmentionedearlier,theselookverysimilarbecausetheybehavealike.However,youmusthavelikelynoticedthatthedefinitionofpip_requirementsitselfdidnothaveanysortofabstraction;thereisnopluggableprovider,novalidation,itdoesn’tsubclasstheResourceclass,amongother

differences.Definitionsprovideyouwithamechanismtodeclarereusablechunksofcodethatyourrecipeswouldotherwiseduplicatesothatyourrecipecanagaindescribethewhat,notthehow.

Thepreviousexampletellsusthatwehaveapip_requirementsobjectandthatwewanttopasssomeparameterstoit,namely,thepathtopip,theuserandgrouptorunpipas,andtherequirements.txtfiletoload.Thesearebroughtintothedefinitionthroughtheparamsargumentandcanbeaccessedasanyothervariabledata.Inthiscase,thedefinitionsaystorunbashasthespecifieduserandgroupandthatthescriptshouldruntheequivalentofthefollowing:

pipinstall-r/path/to/requirements.txt

Thiswillhappenonlyifpipandthe/path/to/requirements.txtfileexist(asindicatedbytheonly_ifguard).Bycreatingsuchadefinition,itcanbereusedanytimeyouneedtoinstallPythonmodulesfromaspecificrequirements.txtfileonyourhost.

RecipesRecipesarewhereallthemagichappenswithChef;theyarethesecretsauce,themanbehindthemask.TheyaretheworkhorsesofconfiguringhostswithChef.RecipesarescriptswritteninRubyusingChef’sDSLthatcontaintheinstructionstobeexecutedonendhostswhentheChefclientisrun.Everytimetheclientisexecutedontheendhost,afewthingshappen:

1. TheendhostmakesarequesttotheChefserversaying,“Ineedtodosomework”.2. TheChefserverlooksattherequestinghost’sidentityanddetermines:

Whichrecipesneedtoberunandinwhatorder(therunlist)Thecomputedconfigurationdataforthathost

3. Thisinformationispassedbacktotheendhostalongwiththenecessaryartifactsitneeds(recipes,templates,andsoon).

4. Theclientthencombinestheconfigurationdatawiththerecipesandbeginstoexecuteitsrunlist.

DevelopingrecipesAsadeveloper,youwillbeplacingyourrecipesinsidetherecipesdirectoryofyourcookbook.Eachrecipeisdesignedtoperformaspecificactionorsetofactionstoachieveagoalsuchasprovisioningaccounts,installingandconfiguringadatabaseserver,customsoftwaredeployments,orjustaboutanyotheractionthatyoucouldperformonaserver.

Akeyconceptwhendevelopingrecipesisthattheyshouldbeidempotent.Forthoseunfamiliarwiththeterm,anidempotentoperationisanoperationthatcanbeappliedmultipletimesandhavethesameoutcome.Considerthefollowingrecipe:

user"smith"do

action:create

systemtrue

comment"AgentSmith"

uid1000

gid"agents"

home"/home/matrix"

shell"/bin/zsh"

Onewouldexpect,fromlookingatthisrecipe,thatChefshouldbeabletoexecuteitonce,fivetimes,oronethousandtimes,anditwouldhavethesameeffectastheinitialapplicationoftherecipe.Therewouldnotbefiveoronethousandusersonthehostwiththeloginnamesmith;therewouldbeonlyonesingleuserwiththeloginnamesmith.Also,inalltheruns,itwouldbeconstructedwiththesameUID1000,thesamegroupname,andsoon.

Similarly,givenaparticularstateofthesystemandassumingnothinghaschangedinbetweenruns,subsequentclientexecutionsshouldproducethesame,consistentendingstate.Inshort,theChefclientshouldbeabletoruntwotimesinarow,andiftheconfigurationhasnotbeenupdated,thesystemshouldlookexactlythesameafterthesecondrunasafterthefirstrun.

Recipesuseprovidedconfigurationdataalongwiththecurrentstateofthehosttodeterminetheflowandactionstakenbythescript.Theexecutionofarecipewilltakethesystemfromitsinitialstate,Sintial,toitsnewstate,Sfinal.Well-writtenrecipesshouldbeidempotentsuchthatifthey’reexecutedimmediatelyafterwardsanynumberoftimeswithnoconfigurationorstatchanges,thenthesystemshouldgofromSfinaltothesameSfinalwithnonewchangestothesystem.Thisallowsyoutokeepyoursystemsinaconsistentstateatalltimes,assumingthatnothinggoeswrongduringtheexecutionofthoseoperations;ifsomethingdoesgowrong,youshouldbeabletoreverttoapreviouslyknowngoodstate.

WritingrecipesAsyouhavealreadyseen,cookbooksprovideawaytocombinerelevantpiecesofconfigurationdatasuchasattributes,templates,resources,providers,anddefinitionsinoneplace.Theonlyreasonthesecomponentsexististosupportourrecipes.Recipescombineresourcesinacertainordertoproducethedesiredoutcome;muchinthesamewayachefwouldcombineingredientsaccordingtohisorherrecipetoproducesomedeliciousfood.Byputtingalloftheseresourcestogether,wecanbuildourownrecipesthatrangefromverysimplesingle-steprecipestomultistep,multiplatformrecipes.

StartingoutsmallAverybasicrecipe,aswehavediscussedbefore,mightonlyleverageoneortworesources.OneofthesimplestconceivablerecipesistheoneweusedearliertoverifythatourChef-soloinstallationwasworkingproperly:

action:create

Hereagain,wearecombiningasingleresource,thefileresource,specifyingthatwewanttocreatethefilenamed$HOME/example.txt,andstorethestring"Greetings$USER"inthatfile.$USERand$HOMEwillbereplacedbytheenvironmentvariables,mostlikelytheloginnameoftheuserthatisexecutingchef-clientandtheirhomedirectoryrespectively(unlesstheenvironmentvariableshavebeentamperedwith).

Followingourgoalofidempotence,executingthisrecipemultipletimesinarowwillhavethesameeffectasonlyrunningitonce.

InstallingasimpleserviceNowthatwe’vecoveredasimplerecipe,let’stakealookatonethatconfigurestheRedisengineandusessupervisordtorunthedaemon.Thisrecipedoesn’tinstallRedis;instead,itdefineshowtoconfigurethesystemtostartandmanagetheservice.Itdoesnothaveanyadvancedlogic,butmerelyconstructssomerequireddirectories,buildsaconfigurationfilefromatemplate,andthenusesthesupervisor_serviceresourcetoconfigurethedaemontorunandbemonitored,asshowninthefollowingcode:

redis_user=node[:redis][:user]

redis_group=node[:redis][:group]

environment_hash={"home"=>"#{node[:redis][:home]}"}

#Createthelogdiranddatadir

[node[:redis][:datadir],node[:redis][:log_dir]].eachdo|dir|

directorydirdo

ownerredis_user

groupredis_group

mode"0750"

recursivetrue

#Generatethetemplatefromredis.conf.erb

template"#{node[:redis][:config_path]}"do

source"redis.conf.erb"

ownerredis_user

groupredis_group

variables({:data_dir=>"#{node[:redis][:data_dir]}"})

mode0644

#Conveniencevariablesforreadability

stdout_log="#{node[:redis][:log_dir]}/redis-stdout.log"

stderr_log="#{node[:redis][:log_dir]}/redis-stderr.log"

redis_bin="#{node[:redis][:install_path]}/bin/redis-server"

redis_conf="#{node[:redis][:config_path]}"

#Tellsupervisortoenablethisservice,autostartit,runitas

#theredisuser,andtoinvoke:

#/path/to/redis-server/path/to/redis.conf

supervisor_service"redis_service"do

action:enable

autostarttrue

user"#{redis_user}"

command"#{redis_bin}#{redis_conf}"

stdout_logfile"#{stdout_log}"

stderr_logfile"#{stderr_log}"

directory"#{node[:redis][:install_path]}"

environmentenvironment_hash

Youwillnoticethatinordertokeeptheconfigurationconsistent,wereusealotofattributes.Forexample,thebeginningoftherecipeusesnode[:redis][:datadir]and

node[:redis][:log_dir]toensurethatthedirectoriesexistbymakinguseofadirectoryresourceinsideofaloop;then,theseareusedlaterontodefinethesupervisorconfigurationvariables(wheretowritelogs)andthetemplatefortheconfigfile(wheretostorethedata).Inall,thisrecipeiscomposedoffourresources:twodirectoriesintheloop,onetemplate,andonesupervisorservice.Bytheendofthisrun,itwillhaveensuredthecriticaldirectoriesexist,Redisisconfigured,andasupervisorconfigurationfileisgenerated(aswellaspokedsupervisordtoreloadthenewconfigurationandstarttheservice).Again,runningthismultipletimes,assumingnoconfigurationchangesinbetweenrunswillputthesystemintheexactsamestate.Rediswillbeconfiguredaccordingtothehostproperties,andsupervisorwillruntheservice.

GettingmoreadvancedLet’smoveupandtakealookataslightlymorecomplicated,yetfairlysimple,recipefromthegitcookbookthatinstallsthegitclientonthehost.Thecookbookismultiplatform,solet’stalkaboutwhatitwillbedoingbeforeitshowsyouthesource.Thisrecipewillbeperformingthefollowingactions:

1. Determinewhichplatformtheendhostisrunningon(byinspectingthenode[:platform]attribute).

2. IfthehostisrunningaDebian-baseddistribution,itwillusethepackageresourcetoinstallgit-core.

3. IfthehostisaRHELdistribution,itwillperformthefollowing:

1. IncludetheEPELrepositorybypullingintheepelrecipefromtheyumcookbookiftheplatformversionis5.

2. Usethepackageresourcetoinstallgit(asthatistheRHELpackagename).

4. IfthehostisWindows,itwillinstallgitviathewindows_packageresourceandinstructittodownloadthefilelocatedatnode[:git][:url](whichinturnpullsfromthedefaultattributesoroverriddenconfiguration),validatethatthechecksummatchestheonespecifiedbynode[:git][:checksum],andtheninstallit;however,thisisonlyiftheEXEisnotalreadyinstalled.

5. IfthehostisrunningOSX,itwillleveragethedmg_packageresourcetoinstalla.pkgfilefroma.dmgimage.Here,thedownloadURL,volumename,packagefile,checksum,andappnameareallattributesthatneedtobeprovided.

6. Finally,ifnoneoftheconditionsaremet,itfallsbacktothepackageresourcetoinstallthegitpackageinthehopethatitwillwork.

Hereisthecodeforthisrecipe:

casenode[:platform]

when"debian","ubuntu"

package"git-core"

when"centos","redhat","scientific","fedora"

casenode[:platform_version].to_i

include_recipe"yum::epel"

package"git"

when"windows"

windows_package"git"do

sourcenode[:git][:url]

checksumnode[:git][:checksum]

action:install

not_if{File.exists?'C:\ProgramFiles(x86)\Git\bin\git.exe'}

when"mac_os_x"

dmg_package"GitOSX-Installer"do

appnode[:git][:osx_dmg][:app_name]

package_idnode[:git][:osx_dmg][:package_id]

volumes_dirnode[:git][:osx_dmg][:volumes_dir]

sourcenode[:git][:osx_dmg][:url]

checksumnode[:git][:osx_dmg][:checksum]

type"pkg"

action:install

package"git"

Onethingwehaven’tseenyetistheuseofthenot_ifqualifier.Thisisexactlywhatitlookslike;iftheblocksuppliedtonot_ifreturnsatruevalue,theresourcewillnotbeprocessed.Thisisveryusefultoensurethatyoudon’tclobberimportantfilesorrepeatexpensiveoperationssuchasrecompilingasoftwarepackage.

SummaryThischapterintroducedyoutothecriticalcomponentsofacookbookthatareusedtowriterecipes.Italsoshowedyousomeexamplerecipestogetyoustarted;thereareanumberofadvancedactionsthatcanbeaccomplishedinyourrecipes,suchassearchingtheChefserverfordata,loadingdatafromdatabags,orusingencrypteddata.Additionally,youcanaddmorecomponentstoyourcookbookssuchascustomresourcesandproviders,tests,andarbitraryRubylibraries.Allofthesewillbediscussedindetailinlaterchapters,butfirstlet’stakealookatwritingsomecompletecookbooks.We’llthenlearnhowtotestthembeforewemoveontolookingatsomecookbooksforcommonsystemadministrationtasks,andthenwe’llprogressontoadvancedtopics.

Chapter5.TestingYourRecipesSofar,youhaveseenhowtomodelyourinfrastructure,provisionhostsinthecloud,andwhatgoesintoacookbook.Oneimportantaspectofdevelopingcookbooksiswritingtestssothatyourrecipesdonotdegradeovertimeorhavebugsintroducedintotheminthefuture.Thischapterintroducesyoutothefollowingconcepts:

UnderstandingtestmethodologiesHowRSpecstructuresyourtestsUsingChefSpectotestrecipesRunningyourtestsWritingteststhatcovermultipleplatforms

Thesetechniqueswillprovetobeveryusefultowriterobust,maintainablecookbooksthatyoucanusetoconfidentlymanageyourinfrastructure.Testsenableyoutoperformthefollowing:

IdentifymistakesinyourrecipelogicTestyourrecipesagainstmultipleplatformslocallyDeveloprecipesfasterwithlocaltestexecutionbeforerunningthemonahostCatchthechangesindependenciesthatwillotherwisebreakyourinfrastructurebeforetheygetdeployedWritetestsforbugstopreventthemfromhappeningagaininthefuture(regression)

TestingrecipesThereareanumberofwaystotestyourrecipes.Oneapproachistosimplyfollowtheprocessofdevelopingyourrecipes,uploadingthemtoyourChefserver,anddeployingthemtoahost;repeatthisuntilyouaresatisfied.Thishasthebenefitofexecutingyourrecipesonrealinstances,butthedrawbackisthatitisslow,particularlyifyouaretestingonmultipleplatforms,andrequiresthatyoumaintainafleetofhosts.Ifyourcookbookruntimesarereasonablyshortandyouhaveasmallnumberofplatformstosupportthem,thenthismightbeaviableoption.Thereisabetteroptiontotestyourrecipes,anditiscalledChefSpec.ForthosewhohaveusedRSpec,aRubytestinglibrary,theseexampleswillbeanaturalextensionofRSpec.IfyouhaveneverusedRSpec,thebeginningofthischapterwillintroduceyoutoRSpec’stestinglanguageandmechanisms.

RSpecRSpecisaframeworktotestRubycodethatallowsyoutouseadomain-specificlanguagetoprovidetests,muchinthesamewayChefprovidesadomain-specificlanguagetomanipulateaninfrastructure.InsteadofusingaDSLtomanagesystems,RSpec’sDSLprovidesanumberofcomponentstoexpresstheexpectationsofcodeandsimulatetheexecutionofportionsofthesystem(alsoknownasmocking).

ThefollowingexamplesinRSpecshouldgiveyouahigh-levelideaofwhatRSpeccando:

#simpleexpectation

it'shouldadd2and2together'do

expect(x).toeq4

#EnsureanyinstanceofObjectreceivesacallto'foo'

#andreturnapre-definedvalue(mocking)

it'verifiesthataninstancereceives:foo'do

expect_any_instance_of(Object)

.toreceive(:foo).and_return(:return_value)

o=Object.new

expect(o.foo).toeq(:return_value)

#Deepexpectations(i.eclientmakesanHTTPcallsomewhere

#insideit,makesureithappensasexpected)

it'shouldmakeanauthorizedHTTPGETcall'do

expect_any_instance_of(Net::HTTP::Get)

.toreceive(:basic_auth)

@client.make_http_call

RSpecandChefSpecAswithmosttestinglibraries,RSpecenablesyoutoconstructasetofexpectations,buildobjectsandinteractwiththem,andverifythattheexpectationshavebeenmet.Forexample,oneexpectsthatwhenauserlogsintothesystem,adatabaserecordiscreated,trackingtheirloginhistory.However,tokeeptestsrunningquickly,theapplicationshouldnotmakeanactualdatabasecall;inplaceoftheactualdatabasecall,amockmethodshouldbeused.Here,ourmockmethodwillcatchthemessageinthedatabaseinordertoverifythatitwasgoingtobesent;then,itwillreturnanexpectedresultsothatthecodedoesnotknowthedatabaseisnotreallythere.

TipMockmethodsaremethodsthatareusedtoreplaceonecallwithanother;youcanthinkofthemasstuntdoubles.Forexample,ratherthanmakingyourcodeactuallyconnecttothedatabase,youmightwanttowriteamethodthatactsasthoughithassuccessfullyconnectedtothedatabaseandfetchedtheexpecteddata.

ThiscanbeextendedtomodelChef’sabilitytohandlemultipleplatformsandenvironmentsverynicely;codeshouldbeverifiedtobehaveasexpectedonmultipleplatformswithouthavingtoexecuterecipesonthoseplatforms.ThismeansthatyoucantesttheexpectationsaboutRedHatrecipesfromanOSXdevelopmentmachineorWindowsrecipesfromanUbuntudesktop,withoutneedingtohavehostsaroundtodeploytofortestingpurposes.Additionally,thedevelopmentcycletimeisgreatlyreducedastestscanbeexecutedmuchfasterwithexpectationsthanwhentheyareperformingsomeworkonanendhost.

Youmaybeaskingyourself,“Howdoesthisreplacetestingonanactualhost?”Theansweristhatitmaynot,andsoyoushoulduseintegrationtestingtovalidatethattherecipesworkwhendeployedtorealhosts.Whatitdoesallowyoutodoisvalidateyourexpectationsofwhatresourcesarebeingexecuted,whichattributesarebeingused,andthatthelogicalflowofyourrecipesarebehavingproperlybeforeyoupushyourcodetoyourhosts.Thisformsatighterdevelopmentcycleforrapiddevelopmentoffeatureswhileprovidingalonger,moreformallooptoensurethatthecodebehavescorrectlyinthewild.

Ifyouarenewtotestingsoftware,andinparticular,testingRubycode,thisisabriefintroductiontosomeoftheconceptsthatwewillcover.Testingcanhappenatmanydifferentlevelsofthesoftwarelifecycle:

Single-modulelevel(calledunittests)Multi-modulelevel(knownasfunctionaltests)System-leveltesting(alsoreferredtoasintegrationtesting)

TestingbasicsInthetest-driven-development(TDD)philosophy,testsarewrittenandexecutedearlyandoften,typically,evenbeforecodeiswritten.Thisguaranteesthatyourcodeconformstoyourexpectationsfromthebeginninganddoesnotregresstoapreviousstateofnon-conformity.ThischapterwillnotdiveintotheTDDphilosophyandcontinuoustesting,butitwillprovideyouwithenoughknowledgetobegintestingtherecipesthatyouwriteandfeelconfidentthattheywilldothecorrectthingwhendeployedintoyourproductionenvironment.

ComparingRSpecwithothertestinglibrariesRSpecisdesignedtoprovideamoreexpressivetestinglanguage.ThismeansthatthesyntaxofanRSpectest(alsoreferredtoasaspectestorspec)isdesignedtocreatealanguagethatfeelsmorelikeanaturallanguage,suchasEnglish.Forexample,usingRSpec,onecouldwritethefollowing:

expect(factorial(4)).toeq24

Ifyoureadtheprecedingcode,itwillcomeoutlikeexpectfactorialof4toequal24.ComparethistoasimilarJUnittest(forJava):

assertEquals(24,factorial(4));

Ifyoureadtheprecedingcode,itwouldsoundmorelikeassertthatthefollowingareequal,24andfactorialof4.Whilethisisreadablebymostprogrammers,itdoesnotfeelasnaturalastheonewesawearlier.

RSpecalsoprovidescontextanddescribeblocksthatallowyoutogrouprelatedexamplesandsharedexpectationsbetweenexamplesinthegrouptohelpimproveorganizationandreadability.Forexample,considerthefollowingspectest:

describeArraydo

it"shouldbeemptywhencreated"do

Array.new.should==[]

ComparetheprecedingtesttoasimilarNUnit(.NETtestingframework)test:

namespaceMyTest{

usingSystem.Collection

usingNUnit.Framework;

[TestFixture]

publicclassArrayTest{

[Test]

publicvoidNewArray(){

ArrayListlist=newArrayList();

Assert.AreEqual(0,list.size());

Clearly,thespectestismuchmoreconciseandeasiertoread,whichisagoalofRSpec.

UsingChefSpecChefSpecbringstheexpressivenessofRSpectoChefcookbooksandrecipesbyprovidingChef-specificprimitivesandmechanismsontopofRSpec’ssimpletestinglanguage.Forexample,ChefSpecallowsyoutosaythingslike:

it'createsafile'do

expect(chef_run).tocreate_file('/tmp/myfile.txt')

Here,chef_runisaninstanceofafullyplannedChefclientexecutiononadesignatedendhost,aswewillseelater.Also,inthiscase,itisexpectedthatitwillcreateafile,/tmp/myfile.txt,andthetestwillfailifthesimulatedrundoesnotcreatesuchafile.

GettingstartedwithChefSpecInordertogetstartedwithChefSpec,createanewcookbookdirectory(hereitis$HOME/cookbooks/mycookbook)alongwitharecipesandspecdirectory:

mkdir-p~/cookbooks/mycookbook

mkdir-p~/cookbooks/mycookbook/recipes

mkdir-p~/cookbooks/mycookbook/spec

Nowyouwillneedasimplemetadata.rbfileinsideyourcookbook(here,thiswillbe~/cookbooks/mycookbook/metadata.rb):

maintainer"Yournamehere"

maintainer_email"you@domain.com"

license"Apache"

description"Simplecookbook"

long_description"Supersimplecookbook"

version"1.0"

supports"debian"

Oncewehavethis,wenowhavethebarebonesofacookbookthatwecanbegintoaddrecipesandteststo.

InstallingChefSpecInordertogetstartedwithChefSpec,youwillneedtoinstallagemthatcontainstheChefSpeclibrariesandallthesupportingcomponents.Notsurprisingly,thatgemisnamedchefspecandcanbeinstalledsimplybyrunningthefollowing:

geminstallchefspec

However,becauseRubygemsoftenhaveanumberofdependencies,theRubycommunityhasbuiltatoolcalledBundlertomanagegemversionsthatneedtobeinstalled.SimilartohowRVMprovidesinterpreter-levelversionmanagementandawaytokeepyourgemsorganized,Bundlerprovidesgem-levelversionmanagement.WewilluseBundlerfortworeasons.Inthiscase,wewanttolimitthenumberofdifferencesbetweentheversionsofsoftwareyouwillbeinstallingandtheversionstheauthorhasinstalledtoensurethatthingsareassimilaraspossible;secondly,thisextendswelltoreleasingproductionsoftware—limitingthenumberofvariablesiscriticaltoconsistentandreliablebehavior.

LockingyourdependenciesinRubyBundlerusesafile,specificallynamedGemfile,todescribethegemsthatyourprojectisdependentupon.Thisfileisplacedintherootofyourproject,anditscontentsinformBundlerwhichgemsyouareusing,whatversionstouse,andwheretofindgemssothatitcaninstallthemasneeded.

Forexample,hereistheGemfilethatisbeingusedtodescribethegemversionsthatareusedwhenwritingtheseexamples:

source'https://rubygems.org'

gem'chef','11.10.0'

gem'chefspec','3.2.0'

gem'colorize','0.6.0'

Usingthiswillensurethatthegemsyouinstalllocallymatchtheonesthatareusedwhenwritingtheseexamples.Thisshouldlimitthedifferencesbetweenyourlocaltestingenvironmentsifyouruntheseexamplesonyourworkstation.

InordertouseaGemfile,youwillneedtohaveBundlerinstalled.IfyouareusingRVM,Bundlershouldbeinstalledwitheverygemsetyoucreate;ifnot,youwillneedtoinstallitonyourownviathefollowingcode:

geminstallbundler

OnceBundlerisinstalledandaGemfilethatcontainsthepreviouslinesisplacedintherootdirectoryofyourcookbook,youcanexecutebundleinstallfrominsideyourcookbook’sdirectory:

user@host:~/cookbooks/mycookbook$>bundleinstall

BundlerwillparsetheGemfileinordertodownloadandinstalltheversionsofthegemsthataredefinedinside.Here,Bundlerwillinstallchefspec,chef,andcolorizealongwithanydependenciesthosegemsrequirethatyoudonotalreadyhaveinstalled.

CreatingasimplerecipeandamatchingChefSpectestOncethesedependenciesareinstalled,youwillwanttocreateaspectestinsideyourcookbookandamatchingrecipe.InkeepingwiththeTDDphilosophy,wewillfirstcreateafile,default_spec.rb,inthespecdirectory.Thenameofthespecfileshouldmatchthenameoftherecipefile,onlywiththeadditionof_specattheend.Ifyouhavearecipefilenameddefault.rb(whichmostcookbookswill),thematchingspectestwouldbecontainedinafilenameddefault_spec.rb.Let’stakealookataverysimplerecipeandamatchingChefSpectest.

WritingaChefSpectestThetest,shownasfollows,willverifythatourrecipewillcreateanewfile,/tmp/myfile.txt:

require'chefspec'

describe'mycookbook::default'do

let(:chef_run){

ChefSpec::Runner.new.converge(described_recipe)

it'createsafile'do

Here,RSpecusesadescribeblocksimilartothewayChefusesaresourceblock(again,blocksareidentifiedbythedo…endsyntaxorcodecontainedinsidecurlybraces)todescribearesource,inthiscase,thedefaultrecipeinsideofmycookbook.Thedescribedresourcehasanumberofexamples,andeachexampleisdescribedbyanitblocksuchasthefollowing,whichcomesfromthepreviousspectest:

it'createsafile'do

Thestringgiventotheitblockprovidestheexamplewithahuman-readabledescriptionofwhattheexampleistesting;inthiscase,weareexpectingthattherecipecreatesafile.WhenourrecipesarerunthroughChefSpec,theresourcesdescribedarenotactuallycreatedormodified.Instead,amodelofwhatwouldhappenisbuiltastherecipesareexecuted.ThismeansthatChefSpeccanvalidatethatanexpectedactionwouldhaveoccurrediftherecipeweretobeexecutedonanendhostduringarealclientrun.

TipIt’simportanttonotethateachexampleblockresetsexpectationsbeforeitisexecuted,soanyexpectationsdefinedinsideofagiventestwillnotfallthroughtoothertests.

BecausemostofthetestswillinvolvesimulatingaChefclientrun,wewanttorunthe

simulationeverytime.Therearetwooptions:executethecodeineveryexampleoruseasharedresourcethatallthetestscantakeadvantageof.Inthefirstcase,thetestwilllooksomethinglikethefollowing:

it'createsafile'do

chef_run=ChefSpec::Runner.new.converge(described_recipe)

Theprimaryproblemwiththisapproachisrememberingthateverytestwillhavetohavetheresourcerunningatthebeginningofthetest.Thistranslatestoalargeamountofduplicatedcode,andiftheclientneedstobeconfigureddifferently,thenthecodeneedstobechangedforallthetests.Tosolvethisproblem,RSpecprovidesaccesstoasharedresourcethroughabuilt-inmethod,let.Usingletallowsatesttodefineasharedresourcethatiscachedforeachexampleandresetasneededforthefollowingexamples.Thisresourceisthenaccessibleinsideofeachblockasalocalvariable,andRSpectakescareofknowingwhentoinitializeitasneeded.

Ourexampletestusesaletblocktodefinethechef_runresource,whichisdescribedasanewChefSpecrunnerforthedescribedrecipe,asshowninthefollowingcode:

let(:chef_run){

Here,described_recipeisaChefSpecshortcutforthenameoftherecipeprovidedinthedescribeblock.Again,thisisaDRY(don’trepeatyourself)mechanismthatallowsustorenametherecipeandthenonlyhavetochangethenameofthedescriptionratherthanhuntthroughthecode.Thesetechniquesmaketestsbetterabletoadapttochangesinnamesandresources,whichreducescoderotastimegoesby.

BuildingyourrecipeTherecipe,asdefinedhere,isaverysimplerecipewhoseonlyjobistocreateasimplefile,/tmp/myfile.txt,ontheendhost:

file"/tmp/myfile.txt"do

owner"root"

group"root"

mode"0755"

action:create

Putthisrecipeintotherecipes/default.rbfileofyourcookbooksothatyouhavethefollowingfilelayout:

mycookbook/

|-recipes/

||-default.rb

|-spec/

|-default_spec.rb

Executingtests

Inordertorunthetests,weusetherspecapplication.ThisisaRubyscriptthatcomeswiththeRSpecgem,whichwillrunthetestscriptsasspectestsusingtheRSpeclanguage.ItwillalsousetheChefSpecextensionsbecauseinourspectest,wehaveincludedthemviathelinerequire'chefspec'atthetopofourdefault_spec.rbfile.Here,rspecisexecutedthroughBundlertoensurethatthedesiredgemversions,asspecifiedinourGemfile,areusedatruntimewithouthavingtoexplicitlyloadthem.Thisisdoneusingthebundleexeccommand:

bundleexecrspecspec/default_spec.rb

ThiswillrunRSpecusingBundlerandprocessthedefault_spec.rbfile.Asitruns,youwillseetheresultsofyourtests,a.(period)forteststhatpass,andanFforanyteststhatfail.Initially,theoutputfromrspecwilllooklikethis:

Finishedin0.17367seconds

1example,0failures

RSpecsaysthatitcompletedtheexecutionin0.17secondsandthatyouhadoneexamplewithzerofailures.However,theresultswouldbequitedifferentifwehaveafailedtest;RSpecwilltelluswhichtestfailedandwhy.

UnderstandingfailuresRSpecisverygoodattellingyouwhatwentwrongwithyourtests;itdoesn’tdoyouanygoodtohavefailingtestsifit’simpossibletodeterminewhatwentwrong.Whenanexpectationinyourtestisnotmet,RSpecwilltellyouwhichexpectationwasunmet,whattheexpectedvaluewas,andwhatvaluewasseen.

Inordertoseewhathappenswhenatestfails,modifyyourrecipetoensurethatthetestfails.Lookinyourrecipeforthefollowingfileresource:

Replacethefileresourcewithadifferentfilename,suchasmyfile2.txt,insteadofmyfile.txt,likethefollowingexample:

file"/tmp/myfile2.txt"do

Next,rerunyourspectests;youwillseethatthetestisnowfailingbecausethesimulatedChefclientexecutiondidsomethingthatwasunexpectedbyourspectest.Anexampleofthisnewexecutionwouldlooklikethefollowing:

[user@host]$bundleexecrspecspec/default_spec.rb

Failures:

1)my_cookbook::defaultcreatesafile

Failure/Error:expect(chef_run).tocreate_file('/tmp/myfile.txt')

expected"file[/tmp/myfile.txt]"withaction:createtobeinChef

run.Otherfileresources:

file[/tmp/myfile2.txt]

#./spec/default_spec.rb:9:in`block(2levels)in<top(required)>'

Finishedin0.18071seconds

1example,1failure

Noticethatinsteadofadot,thetestresultsinanF;thisisbecausethetestisnowfailing.Asyoucanseefromthepreviousoutput,RSpecistellingusthefollowing:

Thecreatesafileexampleinthe'my_cookbook::default'testsuitefailedOurexamplefailedintheninthlineofdefault_spec.rb(asindicatedbythelinethatcontained./spec/default_spec.rb:9)Thefileresource/tmp/myfile.txtwasexpectedtobeoperatedonwiththe:createactionTherecipeinteractedwithafileresource/tmp/myfile2.txtinsteadof/tmp/myfile.txt

RSpecwillcontinuetoexecuteallthetestsinthefilesspecifiedonthecommandline,printingouttheirstatusastowhethertheypassedorfailed.Ifyourtestsarewellwrittenandruninisolation,thentheywillhavenoeffectononeanother;itshouldbesafetoexecuteallofthemevenifsomefailsothatyoucanseewhatisnolongerworking.

ExpandingyourtestsChefSpecprovidesacomprehensivesuiteoftoolstotestyourrecipes;youcanstubandmockresources(replacerealbehaviorwithartificialbehavior,suchasnetworkordatabaseconnections),simulatedifferentplatforms,andmore.Let’stakealookatsomemorecomplexexamplestoseewhatotherthingswecandowithChefSpec.

MultipleexamplesinaspectestSpectestsdonotneedtocontainonlyoneexample;theycancontainasmanyasyouneed.Inordertoorganizethem,youcangroupthemtogetherbywhattheydescribeandsomesharedcontext.InRSpec,contextblockscontainexamplesthatarerelevanttotherecipeorscriptbeingtested.Thinkofthemasself-containedtestsuiteswithinalargertestsuite;theycanhavetheirownresourcesaswellassetupandtear-downlogicthatarespecifictotheteststhatareruninthatcontext.

Asanexample,let’slookatpartofthespectestsuitefromtherender_fileexampleinsideofChefSpecitself.Considerthisportionoftherecipe:

file'/tmp/file'do

content'Thisiscontent!'

cookbook_file'/tmp/cookbook_file'do

source'cookbook_file'

template'/tmp/template'do

source'template.erb'

Therecipebeingshownhasthreeresources:atemplate,acookbook_file,andanordinaryfileresource.Asampleofthematchingspectest(testsremovedforformattingandeaseofreading)containsanouterdescribeblock,whichtellsusthatweareexecutingtestsfortherender_file::defaultrecipeandthreeseparatecontextblocks.Eachcontextdescribesadifferentportionoftherecipethatisbeingtestedandtheexpectationsofthatparticulartypeofresource.Together,theyareallpartofthedefaultrecipe,buttheybehaveverydifferentlyinwhatcontenttheyrenderaswellaswhereandhowtheystorefilesonthesystem.

Inthisexample,thefilecontextcontainsteststhatpertaintotheexpectedresultsofthefileresource,thecookbook_filecontextisconcernedwiththecookbook_fileresource,andsoon:

describe'render_file::default'do

let(:chef_run){

context'file'do

it'rendersthefile'do

expect(chef_run).torender_file('/tmp/file')

expect(chef_run).to_notrender_file('/tmp/not_file')

context'cookbook_file'do

expect(chef_run).torender_file('/tmp/cookbook_file')

expect(chef_run).to_not

render_file('/tmp/not_cookbook_file')

context'template'do

expect(chef_run).torender_file('/tmp/template')

expect(chef_run).to_notrender_file('/tmp/not_template')

Contextscanbeusedtogrouptogetherasetofexamplesthatarerelated,notjustonesthatarespecifictoaparticularresource.Considerthefollowingexample:

describe'package::install'do

context'wheninstallingonWindows2012'do

context'wheninstallingonDebian'do

context'wheninstallingonFreeBSD'do

Inthepreviousexample,ourspectestcontainedteststhataregroupedtogetherbytheplatformbeingexecutedon.Insideofeachcontext,theChefrunwillbeconstructedwithaplatformargumentinsteadsothattheexpectationsbeingtestedwillbeconsideredagainstarunoftherecipeontheplatforminquestionratherthanthehost’soperatingsystem.Thisisincrediblyuseful,aswewillseeinthenextsectionontestingformultipleplatforms.

TestingformultipleplatformsOneofthemorenon-trivialusesofChefSpecistosimulateexecutingrecipesonmultipleplatforms.Thisisusefulfordeveloperswhoarebuildingrecipesthatneedtosupportmorethanoneoperatingsystem.SoftwarepackagessuchasPostgreSQL,MySQL,Java,PHP,Apache,andcountlessotherapplicationscanbeinstalledonmanydifferentplatforms.Becauseeachplatformvariesinitsinstallationmechanism,usercreation,andothercorefeatures,beingabletotestrecipesagainstallthesupportedplatformsisincrediblyuseful.

Let’slookatahypotheticalexampletodeveloparecipetoinstallMySQLonWindows2012andsomethingswemightwanttovalidateduringsucharun:

context'whenrunonWindows2012'do

let(:chef_run)do

#constructa'runner'(simulatechef-client)running

#onaWindows2012host

runner=ChefSpec::ChefRunner.new(

'platform'=>'windows',

'version'=>'2012'

#setaconfigurationvariable

runner.node.set['mysql']['install_path']='C:\\temp'

runner.node.set['mysql']['service_user']='SysAppUser'

runner.converge('mysql::server')

it'shouldincludethecorrectWindowsserverrecipe'do

chef_run.shouldinclude_recipe'mysql::server_windows'

it'shouldcreateanINIfileintherightdirectory'do

ini_file="C:\\temp\\mysql\\mysql.ini"

expect(chef_run).tocreate_templateini_file

Byconstructingtherunnerwiththeplatformandversionoptions,thetestwillexerciserunningthemysql::serverrecipeandpretendasthoughitwererunningonaWindows2012host.Thisallowsustosetupexpectationsaboutthetemplatesthatwillbecreated,recipesthatarebeingexecuted,andmoreonthatparticularplatform.

Presumingthatthemysql::serverrecipewasabletodelegatetotheOS-specificrecipeonagivenplatform,wecouldwriteanothertest:

context'whenrunonDebian'do

let(:chef_run)do

runner=ChefSpec::ChefRunner.new(

'platform'=>'debian'

runner.node.set['mysql']['install_path']='/usr/local'

runner.node.set['mysql']['service_user']='mysql'

runner.converge('mysql::server')

it'shouldincludethecorrectLinuxserverrecipe'do

chef_run.shouldinclude_recipe'mysql::server_linux'

it'shouldcreateanINIfileintherightdirectory'do

ini_file="/usr/local/mysql/mysql.ini"

expect(chef_run).tocreate_templateini_file

it'shouldinstalltheDebianMySQLpackage'do

expect(chef_run).toinstall_package('mysql-server')

Inthisway,wecanwriteourteststovalidatetheexpectedbehavioronplatformsthatwemaynothavedirectaccesstoinordertoensurethattheywillbeperformingtheexpected

actionsforacollectionofplatforms.

SummaryRSpecwithChefSpecextensionsprovidesuswithincrediblypowerfultoolstotestourcookbooksandrecipes.YouhaveseenhowtodevelopbasicChefSpectestsforyourrecipes,organizeyourspectestsinsideofyourcookbook,executeandanalyzetheoutputofyourspectests,andsimulatetheexecutionofyourrecipesacrossmultipleplatforms.

Infuturechapters,wewilllearnsomemoreadvancedtestingmechanismssuchasmockingandstubbingexternalservicessuchassearchanddatabags.Addingtestingtoyourdevelopmentcycleallowsyoutofeelconfidentinthecorrectnessofyourrecipes,whichisacriticalsteptowardsautomatingthemanagementofyourinfrastructure.

Now,let’stakealookathowwewillbuildacookbooktocomplementawebapplicationsothatwecanseethefullcycleofdevelopinganapplicationanddeployingitusingChef.

Chapter6.FromDevelopmenttoDeploymentThischaptercoversend-to-endsoftwaredeploymentofaPython-basedwebapplication.Itwillalsointroduceyoutosomecommoncookbooksandhowtoputthemalltogethertocreateafullyautomateddeploymentmechanism.

Wewillwalkthroughthefollowingtopics:

ConfiguringyourlocalsettingstoworkwithAWSModelingasimpleweb.pyapplicationwithChefInstallingthecookbooksyouneedProvisioningEC2instancesforwebandDBserversDefiningyourrolesAddinguserstohostsInstallingtherequiredsoftwareConfiguringanapplicationusingChefDeployingtheapplication

DescribingthesetupFromahighlevel,hereiswhatneedstohappeninordertotakeanapplicationthatwehavedevelopedfromadesktoptodeployment.Inordertodeployyourapplication,youwillprovisiontwohosts,webanddb(eachwithoneuseraccount)andwebapp,whosehomedirectorywillbein/home/webapp.ThesourcecodewillbehostedonGitHubanddeployedusingGitontothewebserver.Wewillcreateadatabase,provisionanaccounttoaccessthatdatabase,andthenconfigureanddeployaweb.pyapplicationinavirtualPythonenvironmentthatwillbestartedandmonitoredbysupervisord.Thisisafairlycommonpatternformodernwebapplications,regardlessoftheframeworkandlanguagebeingused.Thedemonstrationapplicationusedinthischapterconsistsofonlyahandfuloffilesmakingiteasytodeployandunderstand,butthiswillgiveyoutheconceptsandtoolstoexpandthisexampleforusewithfutureapplicationsyoumightdeveloporneedtodeploy.

DeployingsoftwarewithChefTherearenumerousbenefitstodeployingyoursoftwareusingChef;theprimarybenefitisautomation—thechef-clientcanberunperiodically,anditcanexecutefully-automateddeploymentswheneverchangesaremadetothesourcecoderepository.Additionally,Chefstoresallyourconfigurationdata,soyoucanavoidstoringsensitivesecretsandhard-codingURLsorotherdynamicdatainyourconfiguration.Forexample,ifyouhaveanapplicationwithadatabasepool,andyouaddanewdatabasehosttoyourpool,Chefcanuseasimplesearchtopopulatethelistofhoststoincludeintheconnectionpoolsothatitisalwaysuptodatewithyourinfrastructure.

However,deployingsoftwarewithChefdoesrequiresomecoordinationbetweenyourapplicationandChef.Youwillneedtomaintainrecipesrequiredfordeployingyourapplication,andyouwillalsowanttouseChefastheauthoritativesourceforyourconfigurationdata,whichinvolveswritingconfigurationtemplates.ByusingCheftomanageyourdeployments,youcanalsogenerateanyconfigurationdataneededtorunyoursoftwarebasedonyourinfrastructureconfiguration;inourcase,asimpleconfig.pyfileforyourweb.pyapplication.Thismethodcanalsobeusedtomanagethedatabase.ymlfile(andanyotherYAMLfiles)forRailsapplications,theserver.ymlfileforaDropwizardapplication,oranyotherconfigurationfilesneededtorunyourservice.

TipYAMLisasimplemarkuplanguagetostoreconfigurationdata.Itispopularwithmoderndevelopersbecauseitiseasytoparseandisveryexpressive,similartoJSON.

ConfiguringyourlocalenvironmentBynow,youshouldhaveaccesstoaChefserverofsomesort;here,wewillbeusingthehostedChefservice,butthework(asidefromconfiguringyourknife.rbfile)willremainthesameacrossself-managedandhostedChefinstances.Inordertofollowalongwiththeexamples,youwillneedtoconfigureyourworkstationwithanappropriateknife.rbfileandcertificates.ThesefilescanbedownloadedfromthehostedChefconsoleandmodifiedasneeded.

Additionally,youwillneedtohaveinstalledknifethroughthechefgemandhavetheknife-ec2geminstalledinordertointeractwithEC2.Ifyouprefertouseadifferentprovider,thenyoucanrefertothepreviouschapteronhowtoprovisioncloudhostswiththisproviderwhenyougettotheprovisioningstep.

Inourexample,usinghostedChefandEC2,ourknife.rbfilewillcontaincontentsimilartothefollowingcode:

current_dir=File.dirname(__FILE__)

log_level:info

log_locationSTDOUT

node_name"myorg"

client_key"#{current_dir}/myorg.pem"

validation_client_name"myorg-validator"

validation_key"#{current_dir}/myorg-validator.pem"

chef_server_url"https://api.opscode.com/organizations/myorg"

cache_type'BasicFile'

cache_options(:path=>"#{ENV['HOME']}/.chef/checksums")

cookbook_path["#{current_dir}/../cookbooks"]

knife[:aws_access_key_id]="YOURAWSACCESSKEY"

knife[:aws_secret_access_key]="YOURAWSSECRETKEY"

knife[:region]="AWSREGION"

Again,thebasefilescanbedownloadedfromthehostedChefconsole,orifyouareusingaself-managedChefinstallation,thiscanbefoundonyourChefserver.

ModelingasimplePythonapplicationHerewewillconsideraweb.pyapplicationthathastwoprimarycomponents:awebserverandadatabaseserver.Wewillprovisiononehostforeachrole,bootstrapthem,anddeploythesoftwareontoournewhosts.

Ourapplicationstackwillconsistofthefollowing:

web.pyasourwebframeworkPostgreSQLfordatastorageEC2forvirtualhosts

Wewanttodefinetwoprimaryrolesthatrepresentourwebserverandourdatabaseserver.Inaddition,wewillconstructabaselineroleforallourserversthatwillsupplyanycommondataweneedsuchasuseraccounts,SSHkeys,networkconfigurationdata,shells,commonutilities,libraries,andsoon.

Wewillneedtofindorwritecookbooksforthefollowingcomponentswewilluse:

PythonsupervisordPostgreSQLUseraccountsOurcustomwebapplication

ManagingthecookbooksThecookbooksthatwewillbeusingareallavailableatthefollowingURL:https://github.com/johnewart/simplewebpy_app.Becauseanumberofcookbooksusedinthisexampleareunderactivedevelopment,theonesrequiredfortheexampleshavebeenfrozen(asofwritingofthisbook)toensurecompatibilitywiththeexamples;itisbettertohavethemslightlyoutofdatethanbrokeninthiscase.

However,whenyouwriteyourowncookbooksanddeployyourownsoftwarebeyondthisexample,youwillfindthattherearealargenumberofcookbooksthatcanbefoundthroughtheChefcommunitysite—http://supermarket.getchef.com/—orbysearchingtheWebforcookbooks;manyofthesewillbehostedonGitHub,BitBucket,orsimilarsourcecode-hostingsites.

DownloadingcookbooksHereinthefollowingcode,wewillsimplydownloadthecookbookcollectionasawhole:

http://github.com/johnewart/chef_essential_files

Toinstallthecollection,wecandothefollowingfromthechef_essential_files/cookbooksdirectory:

knifecookbookinstall-o.*

Thiswillinstallallofthecookbooksthatareprovided.Theprovidedcookbooksareallthatisrequiredfortheexamplesinthischaptertobesuccessful.Let’stakealookatourcustomcookbook,thepythonwebappcookbook,asalloftheothersareoff-the-shelfcookbooksthataredesignedtoprovidesomegeneralsupportfunctionalities.

LookingatthedatabaserecipeWewilldoafewthingshere,solet’slookatourdatabaserecipe.Inorderforourwebapplicationtobeuseful,itneedsadatabasetoconnectto.Typically,thisinvolvesinstallingthedatabaseserversoftware,constructingadatabase,andgrantingaccesstothatdatabasebyaspecifieduser(orusers).Ourapplicationisnodifferent,sowewillleveragethedatabasecookbookinordertoaccomplishthis.

First,inourrecipe,weneedtoincludethePostgreSQL-specificresourcesfromthedatabasecookbook,whichwewilldousingthefollowingcode:

include_recipe"database::postgresql"

Youwillneedtoknowwhatdatabaseyouwillbecreatingandtowhichuseryouwillbegrantingaccesstoalongwiththepasswordthatwillbeusedtoidentifythem:

dbname=node[:webapp][:dbname]

dbuser=node[:webapp][:dbuser]

dbpass=node[:webapp][:dbpass]

Inordertocreateadatabaseanduseraswellasgrantaccess,youwillneedtoestablishaconnectiontothedatabaseserverwithauserthathaspermissiontodoso.Youwillseethatthisuserhasalsobeengrantedaccessinyourrole’spg_hbasettingssothatPostgreSQLknowstoallowthepostgresusertoconnecttothedatabaselocally,asshowninthefollowingcode:

postgresql_connection_info={

:host=>'localhost',

:port=>node['postgresql']['config']['port'],

:username=>'postgres',

:password=>node['postgresql']['password']['postgres']

Usingthisconnectioninformation,youcanconstructadatabaseandauser(iftheydon’talreadyexist),andthengrantthatuserfullaccesstoournewdatabase:

#Constructanactualdatabaseontheserver

postgresql_database'webapp'do

connectionpostgresql_connection_info

action:create

#Createadatabaseuserresourceusingourconnection

postgresql_database_userdbuserdo

passworddbpass

action:create

#Grantallprivilegesonalltablesin'webapp'

postgresql_database_userdbuserdo

database_namedbname

privileges[:all]

action:grant

Thishigh-levellanguageallowsustoeasilymanipulatethedatabasewithouttheneedtoknowanydatabase-specificSQLorcommands.IfyouwanttoconvertyourapplicationtouseMySQL,forexample,provisioninganewMySQLdatabasewouldlargelybeaseasyasconvertingthewordpostgresqltomysqlinourrecipe,andthedatabase-specificadapterinthedatabasecookbookwillberesponsiblefortheimplementationdetails.

LookingatyourapplicationdeploymentcookbookOnceourdatabasehasbeenprovisioned,youcanlookathowyoucaninstallourwebapplication.Inthepythonwebapp::webapprecipe,youhavealltheinformationyouneedtodothis.Thewaythatyoudefinearecipefordeployinganapplicationwillvarywildlyamongapplications,aseachapplicationisunique.However,thisparticularexamplewasdesignedtobearepresentativeofmostwebapplications(reasonably)andshouldpresentyouwithagoodstartingpointtounderstandthebasicsofdeployingawebapplicationwithChef.

Modernwebapplicationstypicallyfollowthesamepattern:provisionauser,installaninterpreter,orotherengine(suchasPython,Ruby,andJava),createdirectoriesifneeded,checkoutthesourcecode,runanydatamigrations(ifneeded)toupdateyourdatabase,andthenmakesurethatyourserviceisupandrunning;thisisnodifferent.Themorecomplicatedyourapplication,themoreinfrastructureyoumayneedtomodel,suchasjobqueueengines,asynchronousworkers,andotherlibraries.

Ifyoulookatthewebapplicationcookbooklocatedatcookbooks/pythonwebapp,youwillseethatithasthefollowing:tworecipes,atemplate,andaPIP-requirementdefinitioninsideit.TherecipesincludedareforthewebapplicationitselfandtomanagethecreationofthePostgreSQLdatabaseanduseronthedatabasehost.

Mostoftheinterestingworkisintheapplicationrecipe,cookbooks/pythonwebapp/recipes/webapp.rb;so,let’sstartbytakingalookatthat.Allapplicationsaregoingtohaveaslightlydifferentdeploymentlogic,butmodernwebapplicationsusuallyfollowapatternthatlookslikethefollowing:

Installanysystem-widepackagesrequiredConstructthedirectoriesneededforthesoftwareIfthisisPythonorRuby,possiblyinstallanewvirtualenvtoolorRVMgemsetInstallthelibrariesneededtoruntheapplicationCheckouttheapplication’ssourcecodeBuildandconfiguretheapplicationasneededCreateorupdatethedatabaseschemaStopthewebapplicationservicesStartthewebserverorprocessmanagerthatmonitorstheapplication

Thisexampleapplicationisnodifferent,solet’slookatthestepsneededtodeploythisweb.pyapplication.First,declareanyapplicationconfigurationdataneededwiththefollowingcommand:

app_root=node[:webapp][:install_path]

python_interpreter=node[:webapp][:python]

virtualenv="#{app_root}/python"

virtual_python="#{virtualenv}/bin/python"

src_dir="#{app_root}/src/"

#Grabthefirstdatabasehost

dbhost=search(:node,"role:postgresql_server")[0]['ipaddress']

environment_hash={

"HOME"=>"#{app_root}",

"PYTHONPATH"=>"#{app_root}/src"

Inthissnippet,weusedthecomputedattributestotellourrecipewheretoinstalltheapplication;inthiscase,thedefaultis/opt/webappbutthiscanbeoverriddenforflexibility.Additionally,wesetthepathtothePythoninterpreterwewanttouseforourPythonvirtualenv.However,youcanjustaseasilyspecifyaRubyorJavapathifyourapplicationusedoneofthoselanguages.Thereisapathtothesourcecodeandadatabasehostaddress.ThispathisdeterminedbysearchingtheChefdataforallnodeswiththepostgresql_serverrole,takingthefirstone,andusingitsIP.Thisallowsustoreplacethedatabaseserverandnothavetoworryaboutupdatingourconfigurationdata,whichwe’llseeinabit.

PreparingthedirectoriesInordertodeployourapplication,andforittorun,weneedtohavealocationtoputourdata.Inthisapplication,wehavedefinedaneedfor:aconfigurationdirectory,alogdirectory,andaplacetoviewthesourcecode.Inourrecipe,wewillcreatethesedirectoriesandsetproperownershiptoourdeploymentuserandgroup.Notethatyoudonotneedtocreatetheapplicationrootdirectoryifitalreadyexists,andyoudonotneedtosetspecialownershiporpermissionsontherootdirectory.Becauseweareleveragingtherecursivepropertyofthedirectoryresource,therootapplicationdirectorywillbeimplicitlycreated;however,weareconstructingithereforthesakeofcompleteness.

Itiscriticalthatourdirectorieshavethecorrectownershipandpermissions;withoutthis,theapplicationwillbeunabletointeractwiththosedirectoriestostorelogdataorread-and-writeanyconfigurationdata.Thefollowingcodeconstructsthesedirectoriesforusandchangestheownershipandpermissions:

directory"#{app_root}"do

ownernode[:webapp][:user]

groupnode[:webapp][:group]

mode"0755"

action:create

recursivetrue

#Createdirectories

["src","logs","conf"].eachdo|child_dir|

directory"#{app_root}/#{child_dir}"do

mode"0755"

action:create

recursivetrue

Onethingtonotehereisthatweareusingalooptoconstructourdirectories.Thisisagoodwaytomanagemultipleresourcesofthesametypethathavethesamesetofconfigurationparameters.Herewearesayingthatwehavethreesubdirectories,src,log,andconf.Also,wewanttoconstructadirectoryresourceinsideofourapplication’srootdirectoryforeachsubdirectorywithproperownershipandpermissions.Therecursiveflagissimilartothe-poptiononmkdir,whichtellsittocreateanydirectoriesthataremissinginbetweentherootandthedirectorybeingcreated.

ConstructingyourPythonvirtualenvironmentThismaybenewtonon-Pythondevelopersbutshouldbefairlystraightforward.AvirtualenvironmentoperatesinasimilarwaytoRVMorrbenvforRuby,oraself-containedJARfileforJava.Inthat,itisolatesthePythoninterpreterandinstalledlibrariestoaspecificlocationonthesystem.Inourcase,wewillusethefollowingcodetoachievethis:

python_virtualenv"#{virtualenv}"do

action:create

interpreter"#{python_interpreter}"

Thispython_virtualenvresourcecomesfromthepythoncookbookandwillconstructavirtualenvironmentinthelocationnamedbytheresource(inourcase,thedirectorystoredinvirtualenv,whichaswesawpreviously,isdefinedasthoughinapythondirectoryinsideourapplicationroot)usingthespecifiedinterpreterandownershipproperties.

Avirtualenvironmentwillbecreated,whichcontainsaminimalinstallationofthePythoninterpreteraswellasanyPythonlibrariesthatareinstalledintothevirtualenvironment.Thinkofitasyourapplication’sowninstallationofPythonthatisunaffectedby,andsubsequentlydoesnotaffect,anyotherPythoninstallationonthesystem.

ThisisaveryusefultechniquetoinstallandmanagePythonapplications,andthesameconceptcanbeextendedtotheRailsapplicationusinganysimilartechnologyfromtheRubyworldsuchasRVMorrbenv,asmentionedearlier.

CheckingthesourcecodeOneinterestingthinginthisrecipe,whichhasbeenincludedforfuturereference,istheusageofacookbook,ssh_known_hosts,thatgrabsahost’sSSHkeyandaddsittothesystem’slistofknownSSHkeys.ThisisextremelyusefultodeploysoftwareviaGitHuborBitBucket,whereyouareusingSSHtopulldownthesourcecode,especiallyastheirhostkeysmightchange:

#NeedtoinstallSSHkeyforgithub.com

ssh_known_hosts_entry'github.com'

Notethatitisalsosomewhatinsecureasyouareblindlyacceptingthehost’sfingerprint—ifyouareconcernedaboutsecurity,youcanprovidetheknownfingerprintsmanuallyusingthe:keyattribute.Supplyingthefingerprintisdonethroughthefollowingcode:

ssh_known_hosts_entry'github.com'do

key'github.comssh-rsaAAAAB3NzaC1yc….'

Iftherearealargenumberofhostfingerprintsthatyouneedtomanage,oriftheychangefrequently,youcanuseadatabagtostorethem.Ifyouareinterested,lookattheREADMEforthessh_known_hostscookbookformoreexamples.

OncetheSSHkeysareregistered,youcannowclonethesourcefromagit+sshURLsuchasGitHub’sauthenticatedSSHendpoint.

Inthisexample,weareusingapubliclyavailableHTTPSsourcecoderepository;ifyouweretoreplacethiswithyourownSSH-enabledrepository,youwouldneedtochangetherepositoryattributeandalsomakesuretostoreyourdeploymentkeyontheendhost:

#Cloneourapplicationsource

git"#{src_dir}"do

repository"https://github.com/johnewart/simplewebpy_app.git"

action:sync

branch'master'

usernode[:webapp][:user]

Byusingthegitresource,therepositorywillbeclonedintothedesignatedsourcedirectoryontheendhost.Here,wewillalsobepullingdatafromthemasterbranchandperformingthisactionasourwebappuser.

InstallinganyextradependenciesTherearetwowaystomodeldependenciesforyourapplication:inyourcookbookandrecipe,orthroughanexternalmechanismsuchasBundler,pip,orotherdependencyresolution,andthedownloadingtooldependingonthelanguageofyourchoice.Aswitheverything,therearebothinherentdrawbacksandbenefitstoeachofthesemethods.

ManagingdependenciesinChefBymodelingyourdependenciesinChef,youhaveaconsistentmodelthatyoucanlooktoinacentralizedlocation.ThismeansthatyourapplicationneedsanewRubygem,oraPythonlibrarythatsomeonemustupdateacookbookorChefconfigurationwiththatinformationinorderforthedeploymenttobesuccessful.Thiscanlimityourabilitytocontinuouslydeploybasedsolelyonthecontentsofasourcecoderepository.Ineffect,thisrequiresyoutomodelthefollowinginChef:

DependentlibrariesLibraryversionsPossibly,thedependenciesofanydeclareddependencies(whichcanspiralquickly)

However,modelingitthiswaydoesensurethatChefhasanaccuratepictureofalltheinformationassociatedwithyourapplication.Thissolutiondoesoffersomeotherbenefits:

DependenciesarepreciselymodeledinChefandcanbequeriedbyothertoolsAnysystem-specificpackagesthatareneededforyourinterpretedlibrariesaregoingtobemodeledbyChefanyway,soit’sallinoneplace(examplescanincludenativeXMLordatabaselibraries)Developerscan’tarbitrarilychangedependenciesandaccidentallybreakdeploymentsbecausetheunderlyinglibrarieshavenotbeeninstalledinproduction

Let’slookatsomethingstothinkaboutwhenusingtoolsexternaltoChefforthistask.

ManagingdependencieselsewhereUsinganexternaltoolsuchasBundlerorpiphassomeadvantages,includingflexibilityandeaseofusebydeveloperswhomaynotbeinvolvedininfrastructureconfiguration.Italsointroducesthepossibilityofmisconfigureddependenciesandunderlyinglibraries.Theprimaryadvantageofthismechanismisthatitprovidesasimplerdependencymanagementmodelfordevelopers—simplyaddarequirementtotheGemfile,requirements.txt,orothermetadatafile,andChefwillautomaticallyinstallthemduringthenextrun.Youalsonowhavetolookintwodifferentplacestodeterminewhatisbeinginstalledonendhosts.Thisalsomeansthatyouarenowconfiguringdependenciesinmultipleplaces,increasingthepossibilityofmakingawrongconfigurationchangeinoneplace.

It’simportanttotakeawaythatthereisnotalwaysonlyonetoolforthejob,anddependingonhowyourorganizationorteamoperates,youmaychoosetomixandmatchhowyoumodeltheapplication-leveldependencies.Forthesakeofdemonstratingthembothtoyou,theapplicationcookbookmodelsthedependenciesintherecipeaswellas

througharequirements.txtfileusingpip.Additionally,youmayfindthatinitiallyyourteamusesonewayandthenmovestoanotherasyourrequirementsstopchangingsofrequently,oryouareabletocombinethemtoyouradvantage.

UsingPython’srequirementsfileOurwebappcookbookhasacustompip_requirementsdefinitionthatprovidesaneasywaytoinstallanyrequirementsstoredinsidearequirements.txtfileintoaspecifiedvirtualenvironmentusingthecopyofpipprovidedbythatvirtualenvironment.Inthefollowingcode,youwillseehowwecanachievethis:

pip_requirements"webapp"do

action:run

requirements_file"#{src_dir}/requirements.txt"

Inthisexample,wearetellingpiptorunasourapplication’suserandgroupandtoinstallthedependenciesinourrequirements.txtintothevirtualenvironmentspecifiedbyvirtualenv.Again,asimilarresourcecanbecreated(ifonedoesnotalreadyexist)toexecuteBundlerforRuby,CPANforPerl,orPEARtomanagePHPdependencies.

ConfiguringyourapplicationNowthatyouhavepreparedyoursystemforyourapplication,youneedtoconfigureit.Inorderfortheapplicationtotalktoourdatabase,youmustprovidetherequireddatabaseconnectioninformationthatwehavestoredinChef.Here,wewilluseatemplatethatisstoredintemplates/default/config.py.erb,andinjectitwithourdatabaseconfiguration.Theresourceforthislookslikethefollowingcode:

template"#{src_dir}/config.py"do

source"config.py.erb"

mode"0600"

variables({

:dbname=>node[:webapp][:dbname],

:dbuser=>node[:webapp][:dbuser],

:dbpass=>node[:webapp][:dbpass],

:dbhost=>dbhost

Here,weloadourdatabaseinformationontoourtemplateandstoreitinourapplication’sinstalldirectory(wherewecheckedoutthesourceforsimplicity),andsetsomesanefilepermissions.WerethisaRailsapplication,wecanuseasimilartemplatetogeneratedatabase.ymlandmatchingsettings.yml,orifitwereaDropwizardapplication,aservice.ymlfile,aPHPINIfile,oranyothertypeofconfigurationdatathatwereneeded.Inourcase,wearesimplypopulatingthefollowingPythoncodesothatwehaveadatabaseconnectionobject:

importweb

db_params={

'dbn':'postgres',

'db':'<%=@dbname%>',

'user':'<%=@dbuser%>',

'pw':'<%=@dbpass%>',

'host':'<%=@dbhost%>'

DB=web.database(**db_params)

cache=False

Thepreviousexampleusestheweb.pydatabasemoduletoconstructanewdatabaseconnectionusingthehash,whichcanthenbeimportedandusedintheotherportionsoftheapplication.Again,thisisagoodstartingexampleforourweb.pyapplicationthatcanbeusedasamodelforwhateverframeworkorapplicationserveryouareusinginyoursystems.

KeepingyourapplicationrunningAllapplicationsneedtobestartedandkeptrunninginsomemanner.IfyouareusingRailswithmod_passenger,thentheApachedaemonwillbetheprimaryentrypointforyourapplication,andthissoftwarewillneedtobeinstalledandconfigured.Inourexample,wewillbeusingthesupervisordservicefromhttp://supervisord.org,whichiswritteninPythonandservesasaveryconfigurable,lightweight,andreliableprocessmanager.Youcanconfigureanentryinthesupervisordsystemconfigurationusingasupervisor_serviceresourcethatisprovidedbythesupervisorcookbookinstalledearlier:

supervisor_service"webapp"do

action[:enable,:restart]

autostarttrue

command"#{virtual_python}#{src_dir}/server.py#

{node[:webapp][:port]}"

stdout_logfile"#{app_root}/logs/webapp-stdout.log"

stdout_logfile_backups5

stdout_logfile_maxbytes"50MB"

stdout_capture_maxbytes"0"

stderr_logfile"#{app_root}/logs/webapp-stderr.log"

stderr_logfile_backups5

stderr_logfile_maxbytes"50MB"

stderr_capture_maxbytes"0"

directorysrc_dir

environmentenvironment_hash

Thepreviousexamplewillgenerateaconfigurationfileforsupervisordwiththesettingsspecifiedinourresourceblock.Unlessyouchangethelocation,theconfigurationfileswillbelocatedin/etc/supervisor.d/[service_name].conf.Inourcase,theserviceisnamedwebapp,anditsconfigurationfilewillbe/etc/supervisor.d/webapp.conf.

Here,wearetellingsupervisordthatwewanttoenableourserviceandthenrestartit(whichwillstartitifit’snotcurrentlyrunning),wherewewanttologtheprocess’soutput,howwewanttorotatethoselogfiles,wheretostartourprocess,whatenvironmentvariablestouse,andmostimportantlywhatcommandtoexecute.

Nowthatwe’velookedatourrecipes,let’sgoaheadandsetupourroles,provisionsomesystems,anddeployourapplication!

DefiningrolesHerewewillconstructourthreeroles,oneeachforourbaseserverconfiguration,databaseserver,andwebserver.Eachrolewillhaveasetofrecipestorun,withthebaseserverprovidingtheuseraccounts,SSHkeys,andothercommoncomponents,andthentheothersprovidingconfigurationdataforPostgreSQLandnginx,respectively.

CreatingthebaseserverroleThekeybitsthatareofinterestinourbaseserverrolearetherunlistandtheconfigurationdatathatspecifywhichgrouptopopulateusersfor.Ifyoulookatthefileroles/base_server.json,youwillseethatwehavedefinedonegroupofuserstopullfromourdatabags:

"override_attributes":{

"shell_users":{

"group":"webapp"

Andthen,therecipewewanttousethatwillpopulatetheusersonthehostisintherunlist:

"run_list":[

"users::shell_users"

InordertoloadtheroleintoChef,youcanissueafromfilecommand:

kniferolefromfilebase_server.json

UpdatedRolebase_server!

Youcanverifythattherolewascreatedwithasimplerolelistcommand:

[user]%kniferolelist

base_server

CreatingthedatabaseserverroleLet’stakealookatsomeportionsofourdatabaseserverrole,asdefinedinroles/postgresql_server.json.ThisfilecontainsthedescriptionofourPostgreSQLserverasmodeledearlierinthechapter.Whatisofinterestinthisfileisouroverride_attributessection;thesearesettingswewanttouseinplaceofthedefaultvaluesprovidedbyourpostgresqlcookbook.Asmentionedbefore,youwillwanttolookatthedocumentationandthedefaultattributes.rbfiletofindoutwhatattributesyoucansetforagivencookbookanditsrecipes.

ThePostgreSQLrecipesuseapostgresqlconfigurationsectionthatcontainsaconfigsectionforserver-specificconfigurationandproperties,andpg_hbafortheauthenticationdata.Lookingatthepostgresqlsection,wecanseethatwewanttoinstallVersion9.3,andwewantittolistenonalladdresses(0.0.0.0)onport5432:

"version":"9.3",

"config":{

"listen_addresses":"0.0.0.0",

"port":"5432"

Inaddition,thepg_hbasectioncontainsanarrayofJSONobjectsthatdescribeswhichusershaveaccesstotheservice,bywhatmechanismtheyareabletoauthenticatethemselves(MD5,trusted,localidentservice,andsoon),andfromwheretheycanconnect.Thisiscodedintoourconfiguration,buttherecipescanbeextendedtousedatabagstodeterminethisinformationaswell.Itistoolongtoincludeallofithere,butifyoulookatthepg_hbadata,youwillseethattherearethreeentries:onefortheuserwebusertoconnectfromanywhereusinganMD5password,andtwoforlocaluserstoaccessthedefaulttemplatedatabase,andforthepostgresuseritselftomodifythewebappdatabase.

Inadditiontotheconfigurationdata,thereisarunlist—thistellsChefwhatrecipesthisrolewillinclude.YoucanseefromthisexamplethatwewillbeinstallingthePostgreSQLserverandthenprovisioningourwebapp-specificuseranddatabase(foundincookbooks/pythonwebapp/recipes/database.rb):

"run_list":[

"postgresql::server",

"pythonwebapp::database"

Inordertousethis,wewanttoloadourJSONfileintoCheftodefineourdatabaseserverrole:

kniferolefromfilepostgresql_server.json

UpdatedRolepostgresql_server!

Youcancheckthattherolewascreatedwitharolelistcommand:

base_server

postgresql_server

CreatingthewebserverroleOurwebapplicationroleislocatedintheroles/web_server.jsonfileandcontainstherequiredinformationforourwebserver.IfyoutakealookattheJSONfile,youwillseethattherunlistcontainsfourentries:

"run_list":[

"python",

"python::virtualenv",

"supervisor",

"pythonwebapp::webapp"

BecauseourapplicationreliesonPython,wewanttoinstalltherequiredversionofPythononourhostsaswellasbuildaPythonvirtualenvtoolforourapplication.Inaddition,wewillbeusingsupervisordastheprocessmonitorthatisresponsibleforensuringthatourwebservicestartsandstaysrunning.Wealsoneedtoinstallourwebapplicationoncewehavemetourprerequisites.

SimilartohowweloadedthePostgreSQLrolefromourJSONfile,wecanrepeattheprocessforourwebserverrole:

kniferolefromfileweb_server.json

UpdatedRoleweb_server!

Again,youcancheckthattherolewascreatedwithasimplerolelistcommand:

base_server

postgresql_server

web_server

AddingusersWewillneedarecipetomanageourusers;here,wewillusetheuserscookbook.Wewillcreateoneuser,webuser,whichwillbetheaccountthatisusedfordeploymentanduserconnectivity.WewilldefineouruserinaJSONfilesimilartowhatwedidinthepreviouschapter;placethefollowinginausers/webuser.jsonfile:

"id":"webuser",

"uid":"1000",

"gid":1000,

"shell":"/bin/bash",

"comment":"Webappdeploymentuser",

"groups":["webusers"]

Then,youcanloadthisuserusingthefromfilecommand:

knifedatabagfromfileusersusers

Ensurethatyourhostshaveournewusersbyeditingthebase_serverroleandaddingourwebusersgroupsothatanyusersinthatgroupwillbeprovisionedonallourserversthatincorporatethebase_serverrole:

"shell_users":{

"group":"webusers",

ProvisioningEC2instancesHere,wewillbeprovisioninginstancesinus-west-1,butdependingonwhereyouhaveyourAWSinstancessetup,youwillneedtochangeyourknife.rbconfigurationtospecifytheregionofyourchoice.

Inorderforthemtocommunicatesecurely,wewillconstructasecuritygroupsothatalltrafficbetweenthemispermitted.Thisisoutsidethescopeofthisbook,butitwouldbesomethingtomakesureyouconfigureforproductionsystems,asyouprobablydonotwantthepublicontheInternettohavedirectaccesstoyourdatabaseserver.

Here,wewillassumethatyouhaveyourAWScredentialsandothercriticalcomponentsconfigured,aswecoveredinpreviouschapters.

Toprovisionourdatabaseserver,wewillusethefollowingcommand:

knifeec2servercreate-dubuntu14.04-Iami-ee4f77ab-fm1.small-Zus-

west-1a-Sjewartec2-Ndb00--ssh-userubuntu

Andtoprovisionthewebserver,wewillusethefollowingcommand:

knifeec2servercreate-dubuntu14.04-Iami-ee4f77ab-fm1.small-Zus-

west-1a-Sjewartec2-Nweb00--ssh-userubuntu

Onceyourinstancesareupandrunning,youcannowmoveontoconfiguringthemwiththerolesandconfigurationdatarequired!

ConfiguringthedatabasehostInordertoapplythePostgreSQLroletoourdatabasehost,weneedtomakesureit’sintherunlist.Wecanaccomplishthiswiththefollowingcommand:

knifenoderun_listadddb00"role[base_server]"

knifenoderun_listadddb00"role[postgresql_server]"

Afterensuringthatyournodehasthebase_serverandpostgresql_serverrolesaddedtotherunlist,youcanrunchef-clientonthenewlycreatedhost:

[jewart]%knifessh'name:db00'-xubuntu'sudochef-client'

Oncethisiscomplete,assumingthateverythingwentwell,yournewEC2instancewillhave:

PostgreSQL9.3serverinstalledandrunningAnewdatabase(thenameofwhichisdefinedfromyourconfiguration)AdatabaseuserthatisgrantedpermissiontoconnectCorrectpg_hba.confandpostgresql.conffilesforourservice

Nowthatwehaveourdatabaseserverconfigured,whichcanbeverifiedbyloggingontotheserverandensuringthattheserviceisrunning,let’stakealookatsettingupourwebapplication.

ConfiguringthewebserverInorderforthewebservertodeploythewebapp,weneedtoaddtherequiredrolestothewebserver,aswedidwiththedatabaseserver:

knifenoderun_listaddweb00"role[base_server]"

knifenoderun_listaddweb00"role[web_server]"

Now,wecanexecutechef-clientonthewebhost(againmakingsuretousesudosothatithaspermissiontodoitswork):

[jewart]%knifessh'name:web00'-xubuntu'sudochef-client'

Atthispoint,ourwebserverwillbeinthefollowingstate:

Thefollowingrequiredpackageswillbeinstalled:

Python2.7anddevelopmentlibrariesThePostgreSQLclientdevelopmentlibrariesGit

ThedirectoriesourapplicationneedstorunarecreatedAvirtualenvtool,whichisbasedonthesystemPython2.7iscreatedOurapplicationhasbeencheckedoutfromGitHubAconfigurationfilein/opt/webapp/src/config.pyiscreatedbyChefSupervisordisconfiguredtorunourapplicationandstartstheserver.pydaemon

Now,youshouldbeabletovisityournewlyinstalledwebapplicationatthefollowingURL:http://your-new-ec2-instance-hostname:8080

Ifyoudon’tseeyourapplication,makesurethateachoftheprecedingstepswassuccessful.

DeployingyoursoftwareDeployingsoftwareshouldbetreatedjustlikeyourinfrastructure;repeateddeploymentsofthesamecommitandthesameconfigurationshouldyieldaconsistentstateofyourenvironment.Inthisexample,wewillbeabletodeploynewupdatestoourwebapplicationsimplybyupdatinganynodesthatusetheweb_serverrole.ThecombinationofourrecipeandourconfigurationdatawiththesourcecodehostedinourGitHubrepositorywillensurethatthemostup-to-dateconfigurationandsourceareplacedonourhost.

ManuallydeployingupdatesFuturedeploymentsonlyrequirepushingchangestothemasterbranchandthenrunningchef-clientonanywebserversthatareinthefleet.Thiscanbeaccomplishedonasinglehost(web00)usingthefollowingcommand:

knifessh'name:web00'-xubuntu'sudochef-client'

ThiswilltellknifethatwewanttoSSHtothehostwhosenameisweb00astheubuntuuser(becausethat’sthedefaultEC2userwithsudoaccess)andexecutechef-clientasrootviasudo.Thiswillworkwellifweonlyhaveonehost;however,asyourcapacityincreasesandyouhavemultiplehosts,youwilllikelywanttoexecutethisonagroupofhostsinthefuture.ThiscanbeaccomplishedusingthesearchcapabilityofChefthatallowsyoutoexpandalistofnodesthatmatchasetofcriteria.Here,wewillwanttobuildalistofallthehoststhathaveourweb_serverroleassociatedwiththem.Thefollowingcommandwillaccomplishthis:

knifessh'role:web_server'-xubuntu'sudochef-client'

ThiswilluseChef’ssearchtofindallnodeswiththeweb_serverroleandthenSSHtothemsequentially,thesameasbeforebutonlyacrossmultiplehostsinsteadofjustone.

AutomatingdeploymentThewebapplicationrecipeisdesignedsothatitsyncsthesourcewiththeupstreamGitHubrepository.Bydoingthis,wecanexecutetherecipemultipletimes,andanytimethereareupdates,theywillbepulleddownontothelocalhost.Ifwewantedto,theprocessofdeploymentscouldbeautomatedinthefollowingfashion:

Activedevelopmentoftheapplicationhappensinaseparate,developmentbranchCodeistestedthoroughlyandthenmergedintoamaster(orwhateverbranchisbeingdeployedontohosts)whenitisstableandreadyforproductionHostsareconfiguredtorunchef-clientonafixedintervalusingatoolsuchascronandwillautomaticallyupdatethemselves

Thepossibleissueswiththisarethatbadcodegetsautomaticallydeployedtoendhostsandsoon.However,withenoughintegrationtestingandahighenoughconfidencelevel,ourcodeshouldbesafetodeploytoproductionifitisinthemasterbranch.Throughacombinationoftagsandpropersourcemanagement,rollbackscouldbeassimpleasrevertingthedeploybranchtoaknown-goodtagandtheywouldhappenassoonasthenextchef-clientexecutionorforcedusingknifeasoutlinedpreviously.

SummaryAmajorattractiontoChefandinfrastructureautomationistheabilitytodeploysoftwareandprovisionsystemsquicklyandconsistently.Usingthecookbooksandexamplesoutlinedinthischapter,youshouldbeabletomodelyourapplicationanditscomponents,gathercookbooksrequiredtodeployneededsoftware,buildcookbookstoconfigureanddeploycustomsoftware,andextendtheexamplestoprovidemorefunctionalitiesorenhanceyourinfrastructure.

Nowthatyouhaveseenhowtotakeanapplicationfromdevelopmenttodeployment,let’stakealookatsomemoreadvancedexamplesofcookbookdevelopment,includingwritingcustomprovidersandresources,workingwithsecuredata,searchingChef,andotherwaysofenhancingourrecipesandcookbooks.

Chapter7.BeyondBasicRecipesandCookbooksSofar,wehaveonlyreallylookedathowtousecookbooksasaconsumer,notasaproducer.InordertoharnessthetruepowerofChef,itisimportanttolearnhowtobuildourowncookbooksandrecipesusingthefullfeaturesetthatChefprovides.Thischaptercovers:

AdvancedrecipeconceptsManagingyourdatausingdatabagsSearchingCheffromrecipesAdvancedscriptinginrecipesAuthoringcustomproviders,resources,anddefinitionsDealingwithencrypteddata

ManagingusersBasicusermanagementinChefisachievedthroughtheuseoftheuserresource.Thisresourceallowsyoutoadd,remove,orotherwisemanipulateusersonyourhosts.However,youcan’tpossiblywriterecipesthatcontainoneresourceperuser;itsimplywouldn’tscale.Inordertomakelarge-scaleusermanagementeasier,wecancombinesomeofChef’scapabilitiessuchasdatabags,per-role,per-node,andper-environmentconfigurationtoenablescalableusermanagement.

Let’stakealookatausercookbookthatcanprovidetheseabilities.

EvolutionofashelluserrecipeFirst,let’stakealookataverynaiveusermanagementrecipe.Thiscookbookhasahardcodeduserslist;initially,itcontainsfrodoandsamwiseandsimplyiteratesthroughthelist,creatingusersasitgoes.Hereiswhatthelistmaylooklike:

users=[

'id'=>'frodo',

'uid'=>'100',

'gid'=>100,

'shell'=>'/bin/hobbitshell',

'comment'=>'Frodooftheninefingers'

'id'=>'samwise',

'uid'=>'101',

'gid'=>101,

'shell'=>'/bin/gardenshell',

'comment'=>'Samwisethestrong'

users.eachdo|u|

home_dir="/home/#{u['id']}"

useru['id']do

uidu['uid']

gidu['gid']

shellu['shell']

commentu['comment']

supports:manage_home=>true

homehome_dir

Thisapproachwillworkforahandfulofusers,butithastheproblemofbeingverylimitedinscopeanddifficulttomaintain.Italsoisolatesthelistofuserstothisrecipe,makingitdifficulttoaccessdatafromotherrecipesandverybrittle.Thefirstthingwecanimproveismaketheusersaccessibletothisandanyotherrecipethroughtheuseofdatabags.Let’stakealookathowwecanusedatabagstomakeuserdatamanagementsimplerandmoreflexible.

StoringdataindatabagsDatabagsaredesignedtostorearbitraryconfigurationdatathatpertainstoyourentireinfrastructure.Thismayincludeusers,globalsettings,firewallrules,andsoon;ifitcanbemodeledusingbasicJSONdatastructuressuchasarraysanddictionaries,itcanbeincludedinadatabag.Wehaven’ttouchedmuchontheseyet,sonowisagoodtimetotakealookatwhattheycandowhilemodelingusers.

CreatingadatabagforusersDatabagsarecollectionsofdatathatarerelatedtooneanother;forexample,users,firewallrules,databaseservers,andsoon.Herewewillcreateadatabagthatcontainsouruserdata.ThisisnotintendedtobeareplacementforadirectoryservicesuchasLDAP,thoughyoucouldpotentiallyuseittostoreallyouruserdataandthenwriterecipestopopulateanLDAPserverwithuserdata(inthisway,youmaybeabletokeepanActiveDirectorysystemandaseparateLDAPsysteminsyncbymakingyourChefdatabagtheauthoritativesourceforuserdata).Let’stakealookathowtocreateandmanipulateadatabagwithuserinformation:

[jewart]%knifedatabagcreateusers

Createddata_bag[users]

Now,createanewuser,frodo(youwillneedtohavetheEDITORvariablesettoatexteditorsuchasvimonLinuxsystems):

[jewart]%exportEDITOR=vim

[jewart]%knifedatabagcreateusersfrodo

Databagusersalreadyexists

Youwillbepresentedwithanewentitytemplatethatcontainsonlyonekey,id,whichissettothenameoftheentityyoucreated;inourcase,frodo:

2"id":"frodo",

Savethisfileandyouwillnowhaveone,mostlyempty,entityinyourusersdatabagnamedfrodo.Youcancheckthiswiththeshowsubcommand:

[jewart]%knifedatabagshowusers

Everyiteminadatabaghastohaveauniqueidentifier,whichcanbemeaningfulorjustarandomidentifier;inourcase,itwilldoubleupastheloginnamefortheuser.WecantakeourpreviousdatafromtherecipeandconvertthattodatabagelementsbywritingthemtoJSONfilesanduploadingthemwithknife.Totakeadvantageofuploads,wecancreateadirectory,users,andcreateoneJSONfileperentry:

"id":"frodo",

"uid":"100",

"gid":100,

"shell":"/bin/hobbitshell",

"comment":"Frodooftheninefingers"

"id":"samwise",

"uid":"101",

"gid":101,

"shell":"/bin/gardenshell",

"comment":"Samwisethestrong"

Onceyouhavecreatedthese,youshouldhavetwofiles,frodo.jsonandsamwise.jsoninsideausersdirectory.Inordertobulkuploadthem,weuseaknifedatabagfromthe<dir><databagname>fileinthefollowingmanner:

[jewart]%knifedatabagfromfileusersusers

Updateddata_bag_item[users::frodo]

Updateddata_bag_item[users::samwise]

Youcanverifywhethertheentrieswerecreatedcorrectlywiththeknifedatabagshow<databag><entity_id>command:

[jewart]%knifedatabagshowusersfrodo

comment:Frodooftheninefingers

gid:100

id:frodo

shell:/bin/hobbitshell

uid:100

SearchingfordataNowthatwehaveourdatainadatabaginChef,wecansearchforitusingthesearchcriteria.Forexample,ifwewantedonlyalluserswhosenamesstartwiththeletters,wecansearchwiththefollowingcommand:

[jewart]%knifesearchusers'id:s*'

1itemsfound

chef_type:data_bag_item

comment:Samwisethestrong

data_bag:users

gid:101

id:samwise

shell:/bin/gardenshell

uid:101

Alternatively,ifwewantedalltheusersinagivendatabag,wecanperformthefollowingsearch:

[jewart]%knifesearchusers'id:*'

2itemsfound

data_bag:users

gid:100

id:frodo

uid:100

data_bag:users

gid:101

id:samwise

uid:101

SearchinginsiderecipesNowthatwehavesomedatabagdatacreatedandcanperformbasicsearches,let’sseehowwecanusethattoenhanceourrecipeusingthebuilt-insearchmethod.Thisallowsustoperformthesearcheswejustranwithknifeinsideourrecipes.Thesearchmethodhasasimilarformattotheknifecommand:

search(search_scope,search_criteria)

Thefollowingaresomesimpleexamples:

all_users=search(:users,'id:*')

users_s=search(:users,'id:s*')

all_nodes=search(:node,'*')

Withthis,wecanenhanceourshelluserrecipetousetheentitiesintheusersdatabagratherthanhardcodethem.Ournewrecipewouldlooklikethefollowing:

#Replacethehard-codedusersarraywithasearch:

users=search(:users,'id:*')

#Sameasbefore,we'vejustmovedourdatasource

users.eachdo|u|

useru['id']do

uidu['uid']

gidu['gid']

shellu['shell']

commentu['comment']

supports:manage_home=>true

homehome_dir

Thisisjustasimplesearch;thiswillworkforasmall-scaleinfrastructurewithafixedsetofusers,wherethere’snoneedtorestrictcertaingroupsofuserstocertainhosts.Youcaneasilyimagine,however,asituationwheresomeusersareprovisionedonlytocertainhoststhroughgroups.Let’slookathowwecanachievethiswithsomebetterusermetadataandamoreadvancedsearch.

EnhancingyourusercookbookInourpreviousexample,weusedthesearchmethodtofindalloftheusersinouruser’sdatabag.Herewewillgoonestepfurthertoisolateusersbasedonarbitrarygroupsandseehowwecanlimitthelistofuserstobeprovisionedusingacombinationofsearch,usermetadata,andnodeconfiguration.

First,weneedtoaddagroupskeytoourusers.Let’saddthattoourexistinguserJSONdatafilesandaddafewmoreusers,legolasandgimli:

"id":"frodo",

"uid":100,

"gid":100,

"comment":"Frodooftheninefingers",

"groups":["hobbits","fellowship"]

"id":"gimli",

"uid":201,

"gid":201,

"shell":"/bin/csh",

"comment":"Grumpyolddwarf",

"groups":["dwarves","fellowship"]

"id":"legolas",

"uid":200,

"gid":200,

"shell":"/bin/zsh",

"comment":"KeeneyedLegolas",

"groups":["elves","fellowship"]

"id":"samwise",

"uid":"101",

"gid":101,

"shell":"/bin/gardenshell",

"comment":"Samwisethestrong",

"groups":["hobbits","fellowship"]

Onceagain,weupdatetheexistingrecordsandcreateournewrecordsusingknifedatabagfromfile:

[jewart]%knifedatabagfromfileusersusers

Updateddata_bag_item[users::gimli]

Updateddata_bag_item[users::legolas]

Updateddata_bag_item[users::samwise]

Nowthatyouhaveafewadditionalusersinyourdatabag,andeachuserhassomegroupmetadataattachedtoit,let’stakealookathowwecanusethistoprovisiononlycertainusersonspecifichosts.First,weneedtobeabletolimitoursearchscopedynamically;otherwise,wewillneedtomodifyourrecipeonaper-hostbasisandthatjustwon’tscale.Weneedtoaddadynamicsearchquerytoourrecipewithsomethinglikethefollowingcode:

search_criteria="groups:#{node[:shell_users][:group]}"

Thiscreatesasearchcriteriastringthatwillmatchobjectsthathavethevaluespecifiedsomewhereintheirgroupskey.Inordertomakethisdynamicperhost,wewillstorethisvalueinashell_usershashunderthegroupkey.Forexample,ifyouwantedtoaddallusersthatareinthehobbitsgrouptoaspecificnode,thenyournode’sconfigurationwouldneedtocontainthefollowing:

"shell_users":{

"group":"hobbits",

Thiswillbuildasearchcriteriaof"groups":"hobbits",whichifwepasstothesearchmethodwillyieldallentriesintheusersdatabagthathave"hobbits"insidetheirgroupslist.Considerthefollowingrecipecode:

users=search(:users,search_criteria)

Thenodeconfigurationdatawillexpandthesearchcriteriaduringanexecutiononthisnodetobethefollowing:

search_criteria="groups:hobbits"

Giventhedatawehavestoredinourusersdatabag,thiswouldmatchsamwiseandfrodoastheyhavethehobbitsgroupintheirgroupslist.Wecanverifythisbytryingthesamesearchonthecommandlinewithknife:

[jewart]%knifesearchusers"groups:hobbits"

2itemsfound

data_bag:users

gid:100

groups:

hobbits

fellowship

id:frodo

uid:100

data_bag:users

gid:101

groups:

hobbits

fellowship

id:samwise

uid:101

Asyoucansee,thisallowsustonarrowlydefinethelistofuserstobemanagedthroughthecombinationofentitymetadataanddynamicsearchcriteria.Youcanbuildmoreadvancedapplicationsusingthismethodologywithmoreadvancedsearchcriteriaandincorporatingmoreoftheentities’metadata.

DistributingSSHkeysInadditiontomanaginguseraccounts,wecanalsouseCheftomanageSSHkeys.Becauseagivenuser’sacceptedSSHkeysarestoredinaper-userconfigurationfile,itisquitesimpletomanipulatethem.BycreatingatemplateforSSH-authorizedkeys,wecanbuildarecipethatwilltaketheSSHkeydatafromthedatabagandpopulatetheauthorizedkeysfileonthehost.Bydoingthis,users’SSHkeyscanbestoredinChefanddistributedtoanynumberofhostswithjustonecommand.ThissolvestheproblemstypicallyassociatedwithdistributionandrevocationofSSHkeysinsideanorganization.

TemplatingtheauthorizedkeysHereisasampletemplatewewilluseforouruser’sauthorizedkeysfile;thiswouldbedefinedinanauthorized_keys.erbfile:

<%if@ssh_keys.is_a?(Array)%>

<%=@ssh_keys.join("\n")%>

<%else%>

<%=@ssh_keys%>

<%end%>

Thisisaverysimpletemplatethathasonlytwocases:ifthetemplatevariablessh_keysisanarray,itwillprintthemoutwithanewlineinbetweenthem;otherwise,itwillsimplyprintoutthecontentsofthevariable.

Tousethistemplate,wewillsimplyprovideitwithalistofSSH-compatiblekeystrings:

template"#{home_dir}/.ssh/authorized_keys"do

source"authorized_keys.erb"

owneru['id']

groupu['gid']||u['id']

mode"0600"

variables:ssh_keys=>u['ssh_keys']

Now,wecanmodifyoneofourprevioususerJSONentitiestoaddSSHkeys:

"id":"frodo",

"uid":100,

"gid":100,

"comment":"Frodooftheninefingers",

"groups":["hobbits","fellowship"],

"ssh_keys":[

"ssh-dss

RG9uJ3Qgd29ycnksIFNhbS4gUm9zaWUga25vd3MgYW4gaWRpb3Qgd2hlbiBzaGUgc2VlcyBvbmU

ufrodo@shire",

"ssh-dss

TXkgbWFzdGVyLCBTYXVyb24gdGhlIEdyZWF0LCBiaWRzIHRoZWUgd2VsY29tZS4gSXMgdGhlcmU

gYW55IGluIHRoaXMgcm91dCB3aXRoIGF1dGhvcml0eSB0byB0cmVhdCB3aXRoIG1lPyA=

sauron@mordor"

%knifedatabagfromfileusersssh_keys/frodo.json

Onceyouruserhasbeenupdated,checkwhetheryournewlyaddedmetadatahasbeenupdated,lookingforyournewssh_keyskeyintheentity.Inordertodothat,youcanshowthecontentsofyourdatabagusingthefollowingcommand:

%knifedatabagshowusersfrodo

TheoutputofthisshouldlineupwithyournewlyupdatedJSONcontent.Withtheseadded,wecanwriteanewrecipethatwillallowustodeployauthorized_keysfilesforeachuseronourhosts.OurrecipewillusethesamesearchcriteriafromourpreviousrecipeaswewanttoapplyourSSHkeystoallofourshellusers.

ThisrecipeisresponsibleformakingsurethattheproperdirectoryforSSHiscreatedandhasthecorrectpermissions,aswellascreatingtheauthorized_keysfilewiththenecessarypermissionsandstoringtheSSHkeysassociatedwiththeuserin/home/user/.ssh/authorized_keys:

search_criteria="groups:#{node[:shell_users][:group]}"

search(:users,search_criteria)do|u|

directory"#{home_dir}/.ssh"do

owneru['id']

groupu['gid']

mode"0700"

recursivetrue

template"#{home_dir}/.ssh/authorized_keys"do

source"authorized_keys.erb"

owneru['id']

groupu['gid']

mode"0600"

variables:ssh_keys=>u['ssh_keys']

AddingdeploymentkeysIfyouhaveeverdeployedaRailsapplicationtohoststhatneedtohaveaccesstoyoursourcecodeinaGitHuborBitBucketrepository,thenyouwillknowhowhandyitistomanagedeploymentkeysacrossafleetofhosts.Wecaneasilygeneratearecipethatlooksatanode’slistofdeploymentusersfollowingourpreviousexamplesasastartingpoint.Here,welookfordeployusersinsteadofshellusers,asthesearetheoneswewanttomanagedeploymentkeysfor.Notethatinthisexample,theseuserswouldalsoneedtobeincludedintheshell_usersgrouptoensurethattheygetcreatedbyourpreviousrecipe:

search_criteria="groups:#{node[:deploy_users][:group]}"

search(:users,search_criteria)do|u|

directory"#{home_dir}/.ssh"do

owneru['id']

mode"0700"

recursivetrue

template"#{home_dir}/.ssh/id_rsa"do

source"deploy_key.erb"

owneru['id']

mode"0600"

variables:key=>u['deploy_key']

Tousethisnewrecipe,thedeploymentuserswouldneedtobemodifiedtoincludeagroupidentifierandtheirprivatekey.Thegroupwouldbereservedforusersinvolvedindeployingyourapplicationandbeaddedtotheuser’sgroupskeyinChef.Additionally,anunencryptedSSHprivatekeywouldneedtobepresentinadeploy_keyfield.

TipIncludingunencryptedSSHkeyscanposeasecurityrisk.Thiscanbemitigatedusingencrypteddatabagsoranexternalsecuritymaterialmanagementservice.

WritingcustomextensionsWithChef,youaregivenimmediateaccesstoanumberofresources:files,users,packages,templates,andsoon.However,therewillalwaysbetimeswhenthisdoesnotprovideyouwitheverythingthatyouneed.Fortunately,thebuilt-inChefresourcesorLWRPs(light-weightresourceproviders)arejustRubycodeandwerebuiltwiththeintentionofprovidingaframeworkforenduserstobuildtheirowncomponents.Thismeansthatyoucaneasilybuildyourowncustomresources,andthesecanbesharedwithothersjustlikeanybuilt-inLWRP.

DevelopingacustomdefinitionOneofthesimplestresourcesthatwecanbuildisadefinition—adefinitionislikearesourcewithonlyonebuilt-inprovider.Thesecanbethoughtofasreusablemodulesthatyoucanleverageinsideofyourrecipes.Ifyoufindyourselfwritingthesamethingrepeatedlyinyourrecipes,thenitisprobablyagoodcandidatetowriteacustomdefinition.Forexample,let’slookathowwecanbuildtwodifferentdefinitions:oneforexecutingPython’sPIP,thePythonpackageinstallationtool,toinstallthecontentsofarequirements.txtfileforaPythonapplication,andanothertoinstallapplicationsthatfollowthesamepattern.

OrganizingyourcodeAsdiscussedbefore,cookbookscanhaveadefinitionsdirectoryinsidethem.Thecontentsofthisdirectoryareincludedinyourcookbookrunsandshouldhaveoneperdefinition.ForourPIPresource,wewillcreateafile,definitions/pip_requirements.rb,andforourapplicationtemplate,definitions/python_web_application.rb.Thesewilleachcontaintherespectivedefinitions.

WritingadefinitionforusingPIPDefinitionslooklikeanyChefcomponent—theyarecomposedofresources,variables,scripts,andanythingelseyouuseinarecipe.However,unlikearecipe,theyaredesignedtobereused.Wherearecipeisdesignedwithaspecificeffectinmindsuchasdeployingaspecificapplication,thedefinitionisdesignedtobeconsumedbyrecipestoreduceduplicatecode.

Eachdefinitionisencapsulatedinadefineblock,ano-opversion,orourPIPexamplewouldlooklikethis:

define:pip_requirementsdo

Thisexampledoesabsolutelynothing,butitcanbeusedinarecipeasfollows:

pip_requirements"app_requirements"do

Justinthesamewayyouwoulduseafile,user,ortemplateblockinyourrecipe,youcanuseyourcustomdefinitions.Now,let’senhanceourdefinitionbyusingthenameparameter—thestringargumentpassedtothepip_requirementsblockinyourrecipe;here,itisapp_requirements:

define:pip_requirements,:action=>:skipdo

name=params[:name]

Eachinvocationofadefinitionpassestheparametersintheblocktothedefinition;theseareaccessedinsidethedefinitionthroughtheparamshash.Thereisonespecialparameter,:name,whichcancomefromthefirstargumentbeforetheblock,asshowninthepreviouscode,orfromthenameparameterinsidetheblock.Thisisaconvenienceparameter

designedtomakerecipesmorereadablebyallowingthedevelopertowrite:

resource"somehandydescription"do

Thiscodeiseasiertoreadthan:

resourcedo

name"somehandydescription"

Giventhisinformation,let’slookatthePIPexamplefrompip_requirements.rb:

name=params[:name]

requirements_file=params[:requirements_file]

pip=params[:pip]

user=params[:user]

group=params[:group]

ifparams[:action]==:run

interpreter"bash"

user"#{user}"

group"#{group}"

code<<-EOH

#{pip}install-r#{requirements_file}

only_if{File.exists?("#{requirements_file}")andFile.exists?("#

{pip}")}

Here,thedefinitionexpectsfivearguments:theresourcename,thepathtotherequirements.txtfile,thepipbinarytouse,aswellastheuserandgrouptoexecutepipas.ThereasonthattheresourceacceptsthepathtopipistoallowusingpipinsideaPythonvirtualenvironment.Bydoingthis,thedefinitionbecomesalittlemoreflexibleinsituationswhereyouneedtoinstallyourrequirementsintoadifferentlocationonthesystem.

Alsonotethatwecandefinedefaultparametersaspartofthedefinition’ssignature:

Inthiscase,thedefaultactionis:skip,butitcanbesettoanythingyouwantittobe.Hereitissetto:skipsothatitonlygetsinvokeddeliberatelyratherthanbyvirtueofbeingusedinarecipe.

Asthisisasimpledefinition,itonlycontainsoneresource—ascriptblockthatwilleffectivelyexecutepipinstall-r/path/to/requirements.txtasthespecifieduserandgroup.Anexampleuseofthisdefinitioncanbeseenasfollows:

pip_requirements"requirements"do

action:run

pip"/usr/local/bin/pip"

requirements_file"#{app_root}/src/requirements.txt"

Thiscanbeusedinplaceofthebuilt-inscriptresource:

interpreter"bash"

code<<-EOH

/usr/local/bin/pipinstall-r#{app_root}/src/requirements.txt

only_if{

File.exists?("#{app_root}/src/requirements.txt")and

File.exists?("/usr/local/bin/pip")

FollowingChef’sdeclarativelanguage,buildingdefinitionssuchasthisonemakesitmoreobviousastowhatishappening,ratherthanhowitishappening.Wehaveabstractedtheshellscriptandguardtestsbehindafaçade,thatisthepip_requirementsdefinition,whichismoreclearinitseffectwhenyoureadarecipe;youdon’tneedtoexaminethecontentsofthescriptblocktodeducewhatishappeningastheresourcenametellsyouexactlywhat’sgoingtobedone.

DefiningafullapplicationtemplateIfyouhaveapplicationsthatfollowthesamestructure(thinkapplicationsthatuseacommonframeworksuchasRails,Django,Pyramids,Python-tornado,andsoon),thenyouwouldlikelywanttodefineacommondefinitionforwhatsuchanapplicationlookslike.ConsiderhereadefinitiontoinstallaPythonwebapplicationfromGitHubusingsomecommonidioms:

define:tornado_applicationdo

app_name=params[:name]

app_root=params[:app_root]

app_user=params[:user]

app_group=params[:group]

python_interpreter=params[:python_interpreter]||

"/usr/bin/python3.3"

github_repo=params[:github_repo]

deploy_branch=params[:deploy_branch]||"deploy"

virtualenv="#{app_root}/python"

virtual_python="#{virtualenv}/bin/python"

app_dir="#{app_root}/src/#{app_name}"

#NeedtoinstallSSHkeyforGitHub

#thiscomesfromthessh_known_hostscookbook

ssh_known_hosts_entry'github.com'

#Basepackagerequirements

package"git"

package"libpq-dev"

package"libxml2-dev"

package"python3.3"

package"python3.3-dev"

directory"#{app_root}"do

owner"#{app_user}"

group"#{app_group}"

mode"0755"

action:create

recursivetrue

#Createdirectories

["bin","src","logs","conf","tmp"].eachdo|child_dir|

directory"#{app_root}/#{child_dir}"do

owner"#{app_user}"

group"#{app_group}"

mode"0755"

action:create

recursivetrue

#InstallPythonvirtualenv

python_virtualenv"#{virtualenv}"do

owner"#{app_user}"

group"#{app_group}"

action:create

interpreter"#{python_interpreter}"

#Applicationcheckout

git"#{app_dir}"do

repository"#{github_repo}"

action:sync

user"#{app_user}"

branch"#{deploy_branch}"

#Pythondependenciesforapp

pip_requirements"tornado_app[#{app_name}]"do

action:run

user"#{app_user}"

group"#{app_group}"

requirements_file"#{app_dir}/requirements.txt"

Thisdefinitioncanbeusedasshowninthefollowingexample:

tornado_application"image_resizer"do

app_root"/opt/webapps"

user"webapp"

group"webapp"

deploy_branch"master"

github_repo"git@github.com:myorg/image_resizer.git"

python_interpreter"/usr/bin/python3.3"

Accordingtothepreviousdefinition,thiswoulddothefollowing:

Addasystem-wideSSH-knownkeyforgithub.com(requiredtoperformaGitcloneandguaranteesthatfuturekeychangeswillwork)Installanyrequiredpackagesiftheydidn’talreadyexist,includingGit,Python,andthepostgresqlclientEnsureanyapplication-requireddirectoriesexistfordatasuchasbinaries,logs,configuration,andmoreCreateaPythonvirtualenvironmentbasedonthesuppliedPythoninterpreter(3.3)in<app_root>/python

Cloneorsync(ifitwasalreadycloned)thesourcecodefrom<github_repo>to<app_root>/src/<app_name>

Installtherequirementsspecifiedin<app_root>/src/<app_name>/requirements.txtusingthecopyofpipfromthevirtualenvironmentin<app_root>/python

Assumingyouhadanothersimilarlystructuredapplication,butyouwantedtouseadifferentuser,group,Pythoninterpreter,anddeploymentbranch,youcaneasilyconfigureitusingthefollowingresource:

tornado_application"restful_api"do

app_root"/opt/webapps"

user"restapi"

group"restapi"

deploy_branch"production"

github_repo"git@github.com:myorg/restful_api.git"

python_interpreter"/usr/bin/python3.2"

Asyoucansee,definitionsallowustodefinereusableresourcesinChef.Therearethreeprimarybenefitstothisapproach:

Simplifiedrecipesareeasiertoread,haveclearerintentions,andlesscodetoaudit,whichmakesthemlesserrorproneAnychangestothedefinitionareautomaticallyappliedtoanyusageofthedefinition,whichmeansyoudon’tneedtomaintainmultiplevariationsIt’seasiertotestbecauseit’sdesignedtobeparameterizedandmodular

NowthatyouseehoweasyitistowritecustomresourcesforChefthroughdefinitions,let’sexaminewritingafull-blownresourcethathasaseparateproviderimplementation.

BuildingaresourceAChefLWRPiscomposedoftwoprimarycomponents,aresourceandaprovider.Theresourceistheblueprintforwhatisbeingprovided;itdescribestheresource,includingwhatactionscanbetakenbyaresource,thepropertiesthatdescribetheresource,andanyotherhigh-levelinformationaboutit.Theproviderisresponsiblefortheactualimplementationoftheresource.Inprogrammingterms,theresourceisanabstractclassorinterfacewheretheproviderisaconcreteclassorimplementation.Forexample,oneofChef’sbuilt-inresourcesisthepackageresource;however,thisisaveryhigh-levelresource.Thepackageresourcedescribeswhatapackageisandwhatapackagecandobutnothowtomanagethem.Thatworkislefttotheproviders,includingRPM,APT,FreeBSDpackages,andotherbackendsystemsthatarecapableofmanagingon-diskinstallationofpackages.

DefiningtheresourceAsanexample,let’stakealookatanS3bucketresource:

actions:sync

default_action:syncifdefined?(default_action)#Chef>10.8

#DefaultactionforChef<=10.8

definitialize(*args)

@action=:sync

#TargetfolderonthehosttosyncwiththeS3bucket

attribute:destination,:kind_of=>String,

:name_attribute=>true

#Anythingtoskipwhensyncing

attribute:omit,:kind_of=>Array

#AWSAccess/secretkey

attribute:access_key_id,:kind_of=>String

attribute:secret_access_key,:kind_of=>String

Here,ourresourceisanS3bucketthatdeclarestheactionsitcantakealongwiththeattributesthatitrelieson.Here,ourresourcedeclaresthatithasoneavailableaction,sync,whichisthedefaultactionandthatithasfourattributes:thedestination,whatfilestoskip,theaccesskey,andthesecretkey.

ImplementingtheproviderTheprovideriswherethelogicfortheresourceisplaced—itisresponsibleforactingontheresourcebeingdescribed.ForourS3bucket,itlookslikethefollowing:

require'chef/mixin/language'

#Onlyrunasneeded

defwhyrun_supported?

action:syncdo

Chef::Log.debug("Checking#{new_resource}forchanges")

fetch_from_s3(new_resource.source)do|raw_file|

Chef::Log.debug"copyingremotefilefromorigin#{raw_file.path}to

destination#{new_resource.destination}"

FileUtils.cpraw_file.path,new_resource.destination

new_resource.updated_by_last_action(true)

defload_current_resource

chef_gem'aws-sdk'do

action:install

require'aws/s3'

current_resource=new_resource.destination

current_resource

deffetch_from_s3(source)

protocol,bucket=URI.split(source).compact

AWS::S3::Base.establish_connection!(

:access_key_id=>new_resource.access_key_id,

:secret_access_key=>new_resource.secret_access_key

bucket.objects.eachdo|obj|

name=obj.key

if!new_resource.skip.contains(name)

Chef::Log.debug("Downloading#{name}fromS3bucket#{bucket}")

obj=AWS::S3::S3Object.findname,bucket

file=Tempfile.new("chef-s3-file")

file.writeobj.value

Chef::Log.debug("File#{name}is#{file.size}bytesondisk")

yieldfile

ensure

file.close

Chef::Log.debug("Skipping#{name}becauseit'sintheskiplist")

rescueURI::InvalidURIError

Chef::Log.warn("ExpectedanS3URLbutfound#{source}")

Let’stakealookattheprovider,piecebypiece.Thefirstthingtheproviderdoes,beyondincludinganyrequiredlibraries,istoinformChefthatitsupportswhy-run.ThisisamechanismthatChefprovidessothatresourcescanbemoreeasilytestedbyeffectivelynotwiringaresourcetoaprovider.Thisallowsdeveloperstotesttheirresources,inwhatiseffectivelyadry-runmode,beforerunningthemliveagainstasystem:

#Onlyrunasneeded

defwhyrun_supported?

Next,thereisanactionblock—thisregisterstheprovidedblockasthelogictobeexecutedforthespecifiedaction(inthiscase,:sync).Thishasthegeneralformsuchas:

action:<actionname>do

#Realworkinhere

Inthiscase,theonlysupportedactionissync,andsothereisonlyoneactionblock:

action:syncdo

Chef::Log.debug("Checking#{new_resource}forchanges")

fetch_from_s3(new_resource.source)do|raw_file|

Chef::Log.debug"copyingremotefilefromorigin#{raw_file.path}to

destination#{new_resource.destination}"

FileUtils.cpraw_file.path,new_resource.destination

new_resource.updated

Here,the:syncactionleveragesthefetch_from_s3method,whichyieldsalocalcopyofafileintheremotebucketonceithasbeendownloaded.Then,thefileiscopiedfromthetemporarylocationlocallyintothespecifieddestination.

ModifyingresourcesInsideofthisaction,youwillnoticethatthereisanactor,new_resource(whichisactuallyabuilt-inmethod).Thisdescribeswhatthestateofthenamedresourceshouldbewhentheproviderhascompleteditsexecutionforthespecifiedresource;thismayormaynotdifferfromthecurrentstateoftheresourceonthenode.Inthecaseofaninitialrun,new_resourcewillalmostcertainlybedifferentfromcurrent_resource,butthatmaynotalwaysbethecaseonsubsequentruns.

Asanexample,ifwehavearecipewiththefollowingS3bucketresourcedeclared:

s3_bucket"s3://mychefbucket/.resource"do

action:sync

skip["foo.txt","bar.txt]

destination"/opt/app_data"

access_key_idnode[:app][:aws_access_key]

secret_access_keynode[:app][:aws_secret_key]

ownernode[:app][:user]

mode"0755"

Then,thenew_resourceactorwouldhaveitsmembervariablespopulatedwiththeparameterspassedtothes3_bucketresource.Again,thisistheexpectedstateoftheresource,thewayitshouldbewhentheexecutionbytheprovideriscomplete.Inthiscase,whentheprovidercodeisexecuted,new_resource.destinationwillbe"/opt/app_data"andnew_resource.skipwillbealistof"foo.txt"and"bar.txt"andsoon.ThisallowsyoutopassdataintotheinstanceoftheresourceinthesamewaythatwaspossiblewiththePIPandTornadoapplicationdefinitions.

LoadinganexistingresourceOnethingthatislessobviousabouttheproviderscriptistheload_current_resourcemethodthatisnotcalledfromwithintheprovider.ThismethodisusedbyCheftofindaresourceonthenodebasedontheattributesthatareprovidedbytherecipe.Thisisusefultodetermineifanythingneedstobedonetobringanexistingresourceonthehostsuchasafile,auseraccount,oradirectoryoffiles,uptodatewiththedatathatisprovidedduringexecutionoftherecipe.

Itmightmakesensetoextendthisprovidertoprecomputethehashesofthefilesthatalreadyexistinthedirectoryon-diskasspecifiedbydestination.Thisway,theprovidercanbeupdatedtoonlydownloadanyremotefilesinS3thathaveadifferentfingerprintthanasimilarlynamedresourceondisk.Thispreventsunnecessaryworkfrombeingperformed,whichsavestime,bandwidth,andotherresources.

Here,however,itisalsousedtoensurethatanydependenciestodownloadfilesareinstalled;inthiscase,theAWSgemisrequiredtousetheS3client.Thisworksbecausetheload_current_resourcemethodgetscalledonearlytodeterminethecurrentstateoftheresource.Iftheresourcesarethesame,thentheproviderhasnothingtodo.ThecurrentimplementationjustclobberswhateverfilesarelocalwiththecontentsoftheS3bucket(moreofaone-waydownloadthanasync,really).

DeclaringthataresourcewasupdatedResourceshaveabuilt-inmethod,updated_by_last_action,whichiscalledinsidethe:syncactionallthetimeinthisexample.Thismethodnotifiestheresourcethatthenodewasupdatedsuccessfully.Thisshouldonlybesettotrueifeverythingwassuccessfullyupdated;failuresshouldnotmakethiscallunlesstheysetittofalse.Itisusefultoknowwhatresourceshavebeenupdatedforreportingorotherpurposes.Forexample,youcanusethisflagtoidentifywhatresourceshavebeenupdated:

moduleSimpleReport

classUpdatedResources<Chef::Handler

defreport

Chef::Log.info"Resourcesupdatedthisrun:"

run_status.updated_resources.eachdo|r|

Chef::Log.info"#{r.to_s}"

WorkingwithdatabagsThereareanumberofthingsyoucandowithdatabags.

SecuringyourdatabagsDatabagsarejustJSONdata,buttheyarestoredinthesystemasplaintext,withoutanysecurity.Theyarealsodownloadedontovarioushoststhroughoutthelifecycle,whichcanleadtoleakingofpotentiallysensitiveinformation.Fortunately,Chefhasamethodthatletsyousecurethisdatabyusingknife,alongwithsecretkeystokeepdataindatabagsencrypted.

SecretkeysEncryptingadatabagitemrequiresasecretkey;onewayofgeneratingasecretkeyistogeneratearandomnumberandusetheBase64encodingofthatnumberasthesecretkey.Thisshouldhaveanylineendingsremovedtoensureitworksproperlyonallplatforms,regardlessofplatform-specificlineendings.Hereisaquickwaytogenerateoneusingtheopensslcommandlinetoolcombinedwithtrtoremoveanylineendings:

$opensslrand-base64512|tr-d'\r\n'>~/.chef/data_bag_secret

EncryptingyourdataInordertoencryptyourdatabagitem,youmustuseknifeandpassthe--secretor--secret-fileflagstoknifewhencreatingtheitem.Forexample,tocreateadatabagcalledcredentialsandstoreanewentry,aws,insideit,youwouldusethefollowingcommand(makesureyousetyourEDITORenvironmentvariablefirst):

$knifedatabagcreatecredentialsaws--secret-file

~/.chef/data_bag_secret

Asmentionedbefore,youwillbepresentedwiththecontentsofyournewdatabagiteminyoureditor,unencrypted:

2"id":"aws",·

Here,wecanaddsomeproperties,suchasasecretkey:

2"id":"aws",·

3"secret_key":"A21AbFdeccFB213f"

Onceyousavethis,knifewilltellyouthatthenewdatabagwascreated,alongwiththenewdatabagiteminChef,justasitdidwiththeuserdataearlier.TheonlydifferencewillbethatthistimethedatastoredintheChefserverhasbeenencryptedusingthesymmetrickeyyouprovided:

$knifedatabagcreatecredentialsaws--secret-file

~/.chef/data_bag_secret

Createddata_bag[credentials]

Createddata_bag_item[aws]

Tocheckwhetheryournewlycreateddatabagentrywasencrypted,useknife,aswehave

before,toshowthecontentsofanitem:

$knifedatabagshowcredentialsaws

id:aws

secret_key:

cipher:aes-256-cbc

encrypted_data:

SG4z4jd4VAnJ4gG0wPcJWOX7H+ZNSxG5PH+n7EgHFV9e1SciVznjaAbzK61c

iv:rKB0riCr84QhBkw+Wgc/5Q==

version:1

DecryptingyourdataInordertodecryptthedatainthedatabagitem,youneedtoprovidethesamesymmetrickeyasyouprovidedwhenyouencryptedit,usingthe--secretor--secret-fileargument,ascanbeseenhere:

$knifedatabagshowcredentialsaws--secret-file~/.chef/data_bag_secret

id:aws

secret_key:A21AbFdeccFB213f

Ifitwasn’talreadyobvioustoyou,makecertainyoudonotlosethisfile.Withoutyoursecretkeyorsecretfile,youwillnotbeabletodecryptthedatainyourdatabag.Itmaybeworthencryptingthesecretfilewithapassphraseifyouaregoingtobetransmittingittonontrustedlocationsaswell.

StoringkeysonnodesAnencryptionkeycanalsobestoredinanalternatefileonthenodesthatneedit,andyoucanspecifythepathlocationtothefileinsideanattribute;however,EncryptedDataBagItem.loadexpectstoseetheactualsecretasthethirdargumentratherthanapathtothesecretfile.Inthiscase,youcanuseEncryptedDataBagItem.load_secrettoslurpthesecretfilecontentsandthenpassthem:

#insideyourattributefile:

default[:app][:aws_creds_secret]="/opt/secret/aws.secret"

#Insideyourrecipe

aws_secret=Chef::EncryptedDataBagItem.load_secret

"#{node[:app][:aws_creds_secret]}"

aws_creds=Chef::EncryptedDataBagItem.load

"credentials","aws",aws_secret

aws_creds["secret_key"]

SearchingyourdataAswediscussedearlier,youcansearchthroughyourdatabagsusingBooleansearchlogic.Thispermitsyoutofindonlytheentriesinyourdatabagsthatyouneed.Thesamesearchquerylanguageisusedonthecommandlinewithknifeasitisinyourrecipes,sothatyoucantestyourqueriesonthecommandlinetoensurethattheyproducetherightresultsbeforeyouputtheminyourrecipes.Youcanalsosearchthroughotherresourcesaswell,notjustdatabags.

SearchingyourdatabagswithknifeTheknifetoolusesthesearchcommandtosearchthroughyourdatabags.Thegeneralsyntaxis:

knifesearch<source>"<searchcriteria>"

SearchingyourdatabagsfromarecipeInsidearecipe,thesearchmethodisusedtosearchthroughadatabag.Thesyntaxforthisis:

search(:source,"searchcriteria")

QueryingyourdataThesearchqueryformatisreasonablystraightforwardandlookslikemostothersearchenginesthatsupporttheBooleanlogic.

Searchesonattributescomeintheformofkey:value;soforexample,ifyouwantedtofindalloftheuserswhoweredwarvesfromourearlierdatasets,youcanusethesearchquery:

knifesearchusers"groups:dwarves"

NegatingasearchtermcanbeaccomplishedbyplacingNOTinfrontofthesearchterm.Forexample,alluserswhoarenothobbitswillbe:

knifesearchusers"NOTgroups:hobbits"

YoucanalsouseanORmodifier:

knifesearchusers"groups:elvesORgroups:hobbits"

Thislastsearchcriteriawouldyieldtheuserslegolas,samwise,andfrodoasFrodoandSamwiseareinthegroupcalledhobbitsandLegolasisintheelvesgroup.

Whilecombiningsearchterms,youcanlogicallyANDthemtogetheraswell.Forexample,alluserswithaGIDstartingwith20,whocontainthegroupelvescanbefoundusingthefollowingquery:

knifesearchusers"groups:elvesANDgid:20*"

Youcansearchyournodeswiththesamequerylanguage—inordertofindallnodesthatarerunningsomeformofwindows,youcansearchfortheplatformbeinganythingthat

startswithwin:

knifesearchnode"platform:win*"

ThiswillyieldallWindowshosts(resultshavebeenshortenedabit):

4itemsfound

NodeName:i-13d0bd4f

Roles:

Platform:windows6.2.9200

NodeName:WIN-CJDQ9DEOJFK

Roles:umbraco_cms

NodeName:00c0ff3300

Roles:

NodeName:rs-5889646228538071

Roles:

Or,youcansearchforallnodesthatarerunningwindowsandcontaintheroleumbraco_cms:

knifesearchnode"platform:win*ANDrole:umbraco_cms"

Or,ifyouwantedtoeliminatethosenodesthatruntheUmbracoCMS,youcaneasilyinverttherolecondition:

knifesearchnode"platform:win*ANDNOTrole:umbraco_cms"

BecauseChefusesApacheSolrtosearchitsdata,youcanrefertotheApacheSolrdocumentationonbuildingmoreadvancedquerylogicathttp://wiki.apache.org/solr/SolrQuerySyntax.

ManagingmultiplemachineswithsearchqueriesThesearchcriteriacanbeusedforallsortsofplaces:inrecipes,onthecommandline,throughAPIcalls,andmore.OneveryinterestingapplicationisbeingabletousethesearchquerytoSSHtomultiplemachinestoperformcommandsinparallel:

knifessh"fqdn:*.east.mycorp.comANDplatform:ubuntu""chef-client"-x

app_user

ThiswillcontacttheChefserverandaskforthenodesthatmatchthegivenquerystring(machineswhoseFQDNmatchesthewildcardexpression"*.east.mycorp.com"andthatarerunningUbuntu)andthenconnecttothemviaSSHastheuserapp_userandrunthechef-clientcommandoneachofthem.Again,youcanrestrict(orexpand)theserverlistbyusingamore(orless)specificquery.

Onceyouhavemasteredthisaspectofusingknife,youcanlearnmoreaboutitssupportforexecutingmultipleconnectionsconcurrentlyandeveninteractwithterminalmultiplexerssuchasscreenandtmux.

SummaryChefhaslotsofmechanismstobuildadvancedautomation,includingbuildingyourowndefinitions,resources,andproviders,aswellasstoringandaccessingcomplexconfigurationdataandevensecurelyencryptingit.Thischapterhasshownyouhowtomanagedataindatabags(includingencrypteddata),useChef’sadvancedsearchenginetofindandmanipulatedatainyoursystemfromthecommandlineandinrecipes,aswellasdevelopdefinitionsforreusablerecipedevelopment,andevenbuildcustomresourcesandprovidersforuseinyourcookbooks.

Inthenextchapter,wewillcoversomemoreadvancedwaystouseChef,includinginteractingwiththeChefshell,automationandintegrationwithChefusingscriptsandAPIs,externaltoolsandresources,advancedtestingincludingintegrationtesting,andusingChef-soloandVagranttomanageyourdevelopmentenvironments.

Chapter8.ExtrasYouNeedtoKnowThischapterwillcoverhowtouseCheftobuildcustombootstrapscriptsforsystems,enhancedcommand-linetoolconcepts,leverageChefforautomation,integration,andsecurelystoresensitivedatainthesystem.Sometopicsthatwillbecoveredinthischapterinclude:

UsingChef-solowithVagrantInteractingwiththeChefshellDebuggingrecipesAdvancedcommand-lineusageAutomatingandintegratingwithChefMoretestingmethodologies

VagrantandChef-soloVagrantisaveryusefultooltobuilddevelopmentenvironments,whereitprovidestoolstobuildvirtualmachinesthatcontaineverythingyouneedtogetstartedwithbuildingsoftware.Consider,foramoment,workingonateamthatbuildssoftwareandreliesonaservice-orientedarchitecture(SOA),andthissoftwareiscomposedofanumberofdifferentservices.Inorderforittowork,youmayberequiredtoinstallandconfigureallofthedependentservicestoevenbeginworkingonapartofthesystem;thiscouldbeatime-consuminganderror-proneexerciseforevenseasoneddevelopers.Nowimaginethatallyouhadtodowasdownloadaconfigurationfileandexecutevagranttodoitforyou—thisistheworldofVagrant.

OneoftheinterestingfacetsofVagrantisthatithassupporttoprovisionnewinstancesusinganumberofdifferentmechanisms.Currently,thislistincludes10orsodifferenttools,butthemostinterestingtwoareChef-soloandChefclient.Bynow,youshouldbecomfortablewithhowyoumightprovisionavirtualmachineusingtheChefclient;it’snotmuchdifferentthanprovisioninganEC2instanceoradedicatedserver.However,wehaven’tdiscussedusingChef-solomuchyet,sothisisagoodtimetolearnmoreaboutit.

InstallingVagrantHistorically,VagrantwasinstalledviaRubyGems;thisisnolongerthecase,andifyouhaveanolderversioninstalledasagem,itisrecommendedthatyouremoveitbeforeinstallingVagrant.Installersforallsupportedplatforms(OSX,Windows,andLinux)areavailableatthefollowingURL:

http://www.vagrantup.com/downloads

IfyouarenewtoVagrant,theninadditiontoinstallingVagrant,youwillwanttoinstallVirtualBoxforsimplicity,asVagranthasbuilt-insupportforVirtualBox.VagrantdoessupportotherproviderssuchasVMWareandAWS,butitrequirespluginsthatarenotdistributedwiththecoreVagrantinstallationinorderforthemtowork.

OnceyouhaveinstalledVagrantandVirtualBox,thenyoucancontinueonwiththefollowingexamples.

ProvisioninganewhostwithVagrantProvisioninganewvirtualinstancerequiresthatyoubuildaVagrantconfigurationfilecalledVagrantfile.Thisfileservestwopurposes:todenotethatthedirectoryisaVagrantproject(similartohowaMakefileindicatesaprojectthatisbuiltwithMake),andtodescribethevirtualmachinethatisbeingrun,includinghowtoprovisionit,whatoperatingsystemtouse,wheretofindthevirtualimage,andsoon.Becausethisisjustaplaintextfile,youcanincludeitalongwithanyauxiliaryfilesrequiredtobuildtheimagesuchascookbooks,recipes,JSONfiles,installers,andsoon,andcommitittothesourcecontrolforotherstouse.

Inordertobegin,youwillwanttocreateadirectorythatwillhouseyournewVagrantproject.OnUnix-likesystems,wewouldbootstrapourprojectsimilarlytothefollowingcommand:

mkdir-p~/vagrant/chef_solo

cd~/vagrant/chef_solo

Windowshostswillbethesameexceptfordifferentpathsandchangesinmethodsofdirectorycreation.Oncethisstepiscomplete,youwillneedtocreateaskeletonconfigurationlocatedin~/vagrant/chef_solo/Vagrantfile.Thisfilecanbegeneratedusingvagrantinit,butwewouldnotwanttousethecontentsofthegeneratedfile;so,wewillskipthatstepandmanuallyconstructourVagrantfileinstead(withasimpleone-lineconfigurationthatusesabaseimageofUbuntu13.10).YourVagrantfileshouldlooklikethefollowingcode:

Vagrant.configure("2")do|config|

config.vm.box="ubuntu/trusty64"

Here,"2"istheAPIversion,whichiscurrentlyVersion2.0asofthiswriting,andtheconfiguredbaseimage(orbox)istheUbuntuTrusty(14.04)64-bitimage.Beforeyouusethisbaseimage,itneedstobedownloadedtoyourlocalmachine;youcanaddittoVagrantusingtheboxaddcommand:

vagrantboxaddubuntu/trusty64

TipThisstepwilltakeafewminutesonafastconnection,sobepreparedtowaitwhilethiscompletesifyouarefollowingalonginteractively.Alsonotethatifyouskipthisstep,thebaseimagewillautomaticallybedownloadedwhilerunningvagrantuptostartyourvirtualmachine.

Forfuturereferences,ifyouwanttofindalternativeOSimagestouseforyourVagrantmachines,youshouldlookatVagrantCloud(https://vagrantcloud.com/),whereyoucanfindanumberofotherfreelyavailablebaseimagestodownloadforusewithVagrant.

BootingyourVagrantimageOnceyourbaseimagehascompleteddownloading,youwillusethevagrantupcommandtobootupanewvirtualmachine.Bydoingthis,youwillinstructVagranttoreadtheVagrantfileandbootanewinstanceofthebaseimage:

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'ubuntu/trusty64'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'ubuntu/trusty64'isuptodate…

==>default:SettingthenameoftheVM:

chef_solo_default_1402875519251_51266

==>default:Clearinganypreviouslysetforwardedports…

==>default:Clearinganypreviouslysetnetworkinterfaces…

==>default:Preparingnetworkinterfacesbasedonconfiguration…

default:Adapter1:nat

==>default:Forwardingports…

default:22=>2222(adapter1)

==>default:BootingVM…

==>default:Waitingformachinetoboot.Thismaytakeafewminutes…

default:SSHaddress:127.0.0.1:2222

default:SSHusername:vagrant

default:SSHauthmethod:privatekey

default:Warning:Connectiontimeout.Retrying…

==>default:Machinebootedandready!

==>default:CheckingforguestadditionsinVM…

==>default:Mountingsharedfolders…

default:/vagrant=>/Users/jewart/Temp/vagrant/chef_solo

Asyoucanseefromtheoutput,Vagrantperformedthefollowingthings:

Usedthebaseimageubuntu/trusty64ConfiguredVirtualBoxtouseaNATadapter,mappingport22to2222StartedtheVMinheadlessmode(suchthatyoudon’tseetheVirtualBoxGUI)Createdauser,vagrant,withaprivatekeyforauthenticationMountedasharedfoldermapping/vagrantontheguesttotheVagrantworkspaceonthehost

Nowthatyouhavearunningguest,youcancontrolitbyrunningvagrantcommandsfrominsideofthevagrantworkspace(~/vagrant/chef_solo);forexample,youcanSSHintoitusingthefollowingcommand:

vagrantssh

Andyoucandestroytherunninginstancewiththefollowingcommand:

vagrantdestroy

GoaheadandSSHintoyournewguestandpokearoundalittlebit—youwillnoticethatitlooksjustlikeanyotherUbuntu14.04host.Onceyouaredone,usedestroytodestroyitsothatyoucanlookathowtoprovisionyourVagrantimageusingChef-solo.It’simportanttoknowthatifyouusedestroyonyourguest,changestoyourVagrantimagearenotpersisted;so,anychangesyouhavemadeinsideitwillnotbesavedandwillnot

existthenexttimeyouusevagrantuptostarttheVM.

CombiningVagrantwithChef-soloInourpreviousexample,ourVagrantfilesimplydeclaredthatourguestreliedontheubuntu/trusty64imageasthebaseimageviatheconfig.vm.boxproperty.Next,wewilllookathowtoextendourconfigurationfiletousetheChef-soloprovisionertoinstallsomesoftwareonourguesthost.Here,wewilluseChef-solotoinstallPostgreSQL,Python,andawebapplicationinsideoftheguest.

YouwillprobablynoticethattheconfigurationsectionsintheVagrantfilelooksortoflikeresourcesinChef—thisisbecausetheybothleverageRubyblockstoconfiguretheirresources.SowithVagrant,inordertospecifytheprovisioningmechanismbeingused,theconfig.vm.provisionoptionissettothedesiredtool.Here,wewilluseChef-solo,whichisnamed"chef_solo";so,wewillextendourVagrantfiletoindicatethis:

config.vm.provision"chef_solo"do|chef|

#...Chefspecificsettingsblock

UnderstandingthelimitationsofChef-soloForthemostpart,Chef-solooperatesalotlikethetraditionalclient-servermodeofchef-client.TheprimarydifferencesresultfromthefactthatChef-solodoesnotinteractwithacentralChefserverandthereforelackssupportforthefollowing:

NodedatastorageSearchindexesCentralizeddistributionofcookbooksAcentralizedAPIthatinteractswithandintegratesinfrastructurecomponentsAuthenticationorauthorizationPersistentattributes

Asaresult,ifyouarewritingrecipestobeusedwithChef-solo,youwillbeunabletorelyonsearchfornodes,roles,orotherdataandmayneedtomodifythewayyoufinddataforyourrecipes.Youcanstillloaddatafromdatabagsforcomplexdata,buttheywillnotbecentrallylocated;rather,theywillbelocatedinanumberofJSONfilesthatcontaintherequireddata.

ConfiguringChef-soloThereareanumberofoptionsavailablefortheChef-soloprovisionerinVagrant.Forthemostup-to-datedocumentationofVagrant,besuretovisittheofficialVagrantdocumentationsiteathttp://docs.vagrantup.com/v2/.

MostoftheconfigurationoptionsarewaystoprovidepathstovariousChefresourcessuchascookbooks,databags,environments,recipes,androles.AnypathsspecifiedarerelativetotheVagrantworkspaceroot(wheretheVagrantfileislocated);thisisbecausethesearemountedintheguestunder/vagrantandaretheonlywaytogetdataintothe

hostduringthebootstrapphase.Theoneswewillbeusingare:

cookbooks_path:Thisconsistsofasinglestringoranarrayofpathstothelocationwherecookbooksarestored.Thedefaultlocationiscookbooks.data_bags_path:Thisconsistsofapathtodatabags’JSONfiles.Thedefaultpathisempty.roles_path:ThisconsistsofanarrayorasinglestringofpathswhererolesaredefinedinJSON.Thedefaultvalueisempty.

Inourcase,wewillbereusingourexamplecookbooksfromtheearlierchapter.YoucanfetchthemfromGitHubathttp://github.com/johnewart/chef_cookbook_files;eitherdownloadtheZIPfileorclonethemusingGitlocally.Onceyouhavedonethat,copycookbooks,roles,anddata_bagsfromthearchivetoyourVagrantworkspace.ThesewillbetheresourcesthatyouwilluseforyourVagrantimageaswell.InordertotellVagrant’sChef-soloproviderhowtofindthese,wewillupdateourVagrantfileagaintoincludethefollowingconfiguration:

chef.cookbooks_path="cookbooks"

chef.roles_path="roles"

chef.data_bags_path="data_bags"

TellingChef-solowhattorunInsideoftheprovisionblock,wehaveaChefobjectthateffectivelyrepresentsaChefclientrun.Thisobjecthasanumberofmethods(suchasthepathsettingswealreadysaw),oneofwhichistheadd_recipemethod.Thisallowsustomanuallybuildourrunlistwithoutrequiringrolesordatabagsandcanbeused,asshowninthefollowingexample,toinstallthePostgreSQLserverwithnospecialconfiguration:

#Buildrunlist

chef.add_recipe"postgresql::server"

ThiswilltellVagrantthatwewanttouseourdefineddirectoriestoloadourresources,andwewanttoaddthepostgresql::serverrecipetoourrunlist.Becausecookbooksarebydefaultexpectedtobein[vagrantroot]/cookbooks,wecanshortenthisexampleasshowninthefollowingcode,aswearenotyetusingrolesordatabags:

chef.add_recipe"postgresql::server"

UsingrolesanddatabagswithChef-soloAsyouarealreadyawarebynow,wemaywanttoperformmorecomplexconfigurationofourhosts.Let’stakealookathowtousebothrolesanddatabagsaswellasourcookbookstodeployourPythonwebapplicationintoourVagrantguestsimilartohowwedeployedittoEC2:

#Buildrunlist

chef.add_role("base_server")

chef.add_role("postgresql_server")

chef.add_role("web_server")

Justlikethecookbookspath,therolespathisrelativetotheprojectrootifarelativepathisgiven.

InjectingcustomJSONdata

AdditionalconfigurationdataforChefattributescanbepassedintoChef-solo.ThisisdonebysettingthejsonpropertywithaRubyhash(dictionary-likeobject),whichisconvertedtoJSONandpassedintoChef:

chef.json={

"apache"=>{

"listen_address"=>"0.0.0.0"

Hashes,arrays,andsooncanbeusedwiththeJSONconfigurationobject.Basically,anythingthatcanbeturnedcleanlyintoJSONworks.

Providingacustomnodename

Youcanspecifyacustomnodenamebysettingthenode_nameproperty.Thisisusefulforcookbooksthatmaydependonthisbeingsettosomesortofvalue.Forexample:

chef.node_name="db00"

GettingtoknowtheChefshellTheChefshell,previouslycalledshef,providesaninteractivetoolorread-eval-print-loop(REPL)toworkwithChefresources.MuchinthesamewayIRBoranyotherlanguage’sREPLshellworks,chef-shellisawaytointeractwithknife.Thisishandyforexperimentingwithresourceswhilewritingrecipessothatyoucanseewhathappensinteractivelyratherthanhavingtouploadyourcookbooktoaserverandthenexecutingthechef-clientonatargetnode.Additionally,theChefshellprovidesaresourcetoaddbreakpointstorecipeexecutionsothatitcanbeusedtodebugrecipeexecution,whichisaveryhandyfeature.

UsingtheChefshellAsof11.x,shefhasbeenreplacedwithchef-shellandcanbeusedinthreedifferentmodes:standalone,solo,andclientmode.Eachofthesehasaslightlydifferentsetoffunctionalitiesandexpectedusecases.

ThestandalonemodeThestandalonemodeisusedtorunChefinaninteractivemodewithnothingloaded;thisisalmostlikerunninganREPLsuchasirborpythononthecommandline.Thisisalsothedefaultbehaviorofchef-shellifnothingisspecified.

ThesolomodeThesolomodeisinvokedusingthe-sor--solocommand-lineflagandisawaytousechef-shellasachef-soloclient.Itwillloadanycookbooksusingthesamemechanismthatchef-solouserswould,anditwilluseanychef-soloJSONfileprovidedtoitusingthe-jcommand-lineoption.

Thefollowingareexamplesofusingthesolomode:

chef-shell-s

chef-shell-s-j/home/myuser/chef/chef-solo.json

TheclientmodeTheclientmodeisenabledwiththe-zor--clientcommand-lineflag;thismodecauseschef-shelltoactasthoughyouinvokedchef-clientonthehost.Theshellwillreadthelocalclientconfigurationandperformthenormaldutiesofchef-client:connectingtoyourChefserveranddownloadinganyrequiredrunlists,attributes,andcookbooks.However,itwillallowforinteractiveexecutionsothatitispossibletodebugordiagnoseissueswithrecipesontheendhost.Whenusingtheclientmode,youcanuseanalternateconfigurationfilewiththe-ccommand-lineoption,orspecifyadifferentChefserverURLviathe-scommand-lineoption.

Theexampleusesthefollowing:

chef-shell--client-c/etc/chef/alternate.conf

chef-shell--client-shttp://test.server.url:8080/

InteractingwiththeChefserverusingtheshellTheChefshellprovidesyouwiththeabilitytointeractwiththeserverquicklyinthesamewayyouwoulduseknife,butwithouttheoverheadoftypingknifesearchnode…orknifenodelist,andsoon.ItisaveryconvenientwaytoquerythedatastoredintheChefserverinteractively.Inordertointeractwiththeserverfromyourworkstation,youneedtomakesurethatyourshell’sconfigurationfile,locatedin~/.chef/chef_shell.rb,isconfiguredproperly.Ifyouareconnectingwithchef-shellfromanode,thentheconfigurationin/etc/chef/client.rb(orsimilaronWindows)willbeusedinstead.

Thisfile,similartotheknife.rborclient.rbfile,containstherequiredcertificatedataandconfigurationdatatoconnecttotheChefserver.Anexampleconfigurationfilewillresemblethefollowing,withpaths,organization,andclientnamesupdatedaccordingly:

node_name'myorg'

client_keyFile.expand_path('~/.chef/client.pem')

validation_keyFile.expand_path('~/.chef/validator.pem')

validation_client_name"myorg-validator"

chef_server_url'https://api.opscode.com/organizations/myorg'

Allofthesefilesarepresentifyourknifeinstallationisoperational,andtheconfigurationfilecloselyresemblesthatofknife.rb—ifyouneedvaluesfortheseonyourworkstation,takealookatthe~/.chef/knife.rbfile.Onceyouhaveconfiguredyourshell,youcanpassthe-zcommand-lineflagtoconnectasthechef-clientwould:

[jewart]%chef-shell-z

loadingconfiguration:/Users/jewart/.chef/chef_shell.rb

Sessiontype:client

Loading…...resolvingcookbooksforrunlist:[]

SynchronizingCookbooks:

Thisisthechef-shell.

ChefVersion:11.12.8

http://www.opscode.com/chef

http://docs.opscode.com/

run'help'forhelp,'exit'or^Dtoquit.

Ohai2ujewart@!

InteractingwithdataFromhere,youcaninteractwiththeChefserverinavarietyofways,includingsearching,modifying,anddisplayinganydataelements(roles,nodes,databags,environments,cookbooks,andclients),performingaclientrun(includingsteppingthroughit,onestepatatime),assumingtheidentityofanothernode,andprintingtheattributesofthelocalnode.Forexample,listingtherolesontheChefservercanbeperformedwiththeroles.allmethod,shownasfollows:

chef>roles.all

=>[role[umbraco_cms],role[umbraco],role[base_server],role[web_server],

role[postgresql_server]]

SearchingyourdataSearchingthedataelementsisalsosupported,aseachdatatypehasafindmethodattachedtoit.Thefindmethodtakesamapoftheattributeandpatterntolookforandreturnstheresults.Forexample,youcanfindallrolesontheChefserverthatbeginwith"um"withthefollowingcommand:

chef>umbraco_roles=roles.find(:name=>"um*")

=>[role[umbraco_cms],role[umbraco]]

EditingyourdataAnyobjectinyourChefservercanbeediteddirectlyfromtheChefshellusingtheeditcommandfrominsidetheshell.ThiswillinvokeyourfavoriteeditortoedittherawJSONoftheobjectinquestion,whichprovidesamoredirectmechanismoverusingknife(node|role|databag)editonthecommandline,asyoucanquicklymanipulateanumberofrecordsalotmoreeasily.Forexample,toeditalloftherolesthatcontainthename"apache"andsavetheresults,youcanusethefollowingRubycode:

chef>apache_roles=roles.find(:name=>"*apache*")

>[...somelist…]

chef>apache_roles.eachdo|r|

chef>updated=editr

chef>updated.save

chef>end

Thiswillfindallroleswhosenamecontains"apache".Thenforeachrecord,edittheJSON,storingtheresultsinvariablenamedupdated,andthensavethatrecordbacktotheChefserver.

TransformingdataInthisway,youcaninteractwithanyoftheresourcesthatareavailabletoyou,allowingyoutoquicklyfindandmanipulateanydatastoredinChefdirectlyusingtheRubycode.Forexample,tofindallclientswithagivenstringintheirnameanddisabletheiradministrativeaccess,youcanusethefollowingcode:

clients.transform("*:*")do|client|

ifclient.name=~/bad_user/i

client.admin(false)

TipUsecautionwhentransformingyourdatafromtheChefshell;itisanincrediblypowerfultool,butitseffectsaredestructive.Thesechangesarenotreversible(atleastnotwithoutforethoughtorbackups)andcoulddamageyourdataifyouarenotcareful.Forexample,

ifthepreviouscodewastranscribedincorrectly,itcouldpotentiallyrenderallusersunabletoadministerthesystem.

ExecutingrecipeswithChefshellTwogreatfeaturesofchef-shellaretheabilitytorewindarun(tostepbackwards)andtobeabletostepforwardintherunoneresourceatatime.Asanexample,let’slookathowtodefineasimplerecipeinchef-shellinteractivelyandthenrunit,startitover,andstepthroughit.

First,let’sfireupchef-shellwiththefollowingcommand:

[jewart]%chef-shell

loadingconfiguration:none(standalonesession)

Sessiontype:standalone

Loading…...done.

Thisisthechef-shell.

ChefVersion:11.12.8

http://www.opscode.com/chef

http://docs.opscode.com/

run'help'forhelp,'exit'or^Dtoquit.

Ohai2ujewart@!

Thechef-shellpromptwillchangebasedonthestateyouarein.Ifyouareworkingwitharecipe,thepromptwillchangetobechef:recipe>.

CreatingarecipeintheshellTheChefshellhasanumberofmodes—recipemodeandattributemode.Recipemodeisactivatedwhenworkingwithrecipesandwillbewhatweusehere.Inordertoactivateit,typerecipe_modeattheprompt:

chef>recipe_mode

chef:recipe>

Here,wewillcreateresourcestocreateafileinthecurrentdirectoryinteractivelyusingafileresourcewithnoassociatedconfigurationblock,onlythename:

chef:recipe>file"foo.txt"

=><file[foo.txt]@name:"foo.txt"@noop:nil@before:nil@params:{}

@provider:Chef::Provider::File@allowed_actions:[:nothing,:create,

:delete,:touch,:create_if_missing]@action:"create"@updated:false

@updated_by_last_action:false@supports:{}@ignore_failure:false

@retries:0@retry_delay:2@source_line:"(irb#1):1:in'irb_binding'"

@guard_interpreter::default@elapsed_time:0@resource_name::file@path:

"foo.txt"@backup:5@atomic_update:true@force_unlink:false

@manage_symlink_source:nil@diff:nil@sensitive:false@cookbook_name:

nil@recipe_name:nil>

TipOnethingtonotehereisthattheshellwillprintouttheresultsofthelastoperationexecutedintheshell.ThisispartofanREPLshell’simplicitbehavior;itistheprintpart

ofREPL:inputisreadandevaluated,thentheresultsareprintedout,andtheshellloopstowaitformoreinputfromtheuser.Thiscanbecontrolledbyenablingordisablingtheechostate;echooffwillpreventtheprintedoutputandechoonwillturnitbackon.

Itiscriticaltonotethat,atthispoint,nothinghasbeenexecuted;wehaveonlydescribedafileresourcethatwillbeacteduponiftherecipeisrun.Youcanverifythisbymakingsurethatthereisnofilenamedfoo.txtinthedirectoryyouexecutedchef-shellfrom.Therecipecanberunbyissuingtherun_chefcommand,whichwillexecuteallofthestepsintherecipefromstarttofinish.Hereisanexampleofthis:

chef:recipe>run_chef

INFO:Processingfile[foo.txt]actioncreate((irb#1)line1)

DEBUG:touchingfoo.txttocreateit

INFO:file[foo.txt]createdfilefoo.txt

DEBUG:foundcurrent_mode==nil,sowearecreatinganewfile,updating

DEBUG:foundcurrent_uid==nil,sowearecreatinganewfile,updating

DEBUG:foundcurrent_gid==nil,sowearecreatinganewfile,updating

DEBUG:foundcurrent_uid==nil,sowearecreatinganewfile,updating

INFO:file[foo.txt]ownerchangedto501

DEBUG:foundcurrent_gid==nil,sowearecreatinganewfile,updating

INFO:file[foo.txt]groupchangedto20

DEBUG:foundcurrent_mode==nil,sowearecreatinganewfile,updating

INFO:file[foo.txt]modechangedto644

DEBUG:selinuxutilitiescannotbefound.Skippingselinuxpermission

fixup.

DefiningnodeattributesJustasinanyrecipe,attributescanbeusedintherecipesdefinedintheshell.However,inthestandalonemode,therewillbenoattributesdefinedinitially;soloandclientmodeswilllikelyhaveattributesdefinedbytheirJSONfileortheChefserver,respectively.Inordertointeractwiththecurrentlydefinedattributes,wemustswitchbetweentherecipemodeandattributemode.Thisisachievedusingtheattributes_modecommandasshowninthefollowingcode:

chef:recipe>attributes_mode

chef:attributes>

Herewecanperformtwoprimaryoperations:gettingandsettingnodeattributes.ThesearewaysofmodifyingthevaluesthatareaccessedfrominsidethenodeMashinarecipe.

TipRememberthatthenode’sattributesareaccessedasaMash,akey-insensitivehashthatallowsyoutointerchangestringkeyswithsymbolkeys.TheMashclassisnotabuilt-in

structureinRuby—itisprovidedbyChefforconveniencesothathashkeyscanbeeithersymbolsorstringsandhavethesameeffect.

Settingattributes

Settingattributesisachievedusingthesetcommand,whichhasthefollowingform:

set[:key]=value

Here,:keycanbeasingle-levelkeyoramultilevelkeysimilartoanyentryintheattributes/default.rbfile.Asanexample,wecanconstructanapplicationconfigurationusingthefollowing:

set[:webapp][:path]="/opt/webapp"

set[:webapp][:db][:username]="dbuser"

set[:webapp][:db][:password]="topsecret"

set[:webapp][:user]="webuser"

set[:postgresql][:config][:listen]="0.0.0.0"

Anyparentkeysthatarenon-existentareimplicitlycreatedonthefly,soyoudonotneedtodosomethinglikethefollowing:

set[:webapp]={}

set[:webapp][:path]="/opt/webapp"

Accessingattributes

Inordertodisplayanattributewhenintheattributesmode,simplytypeinthenameofthekeyyouareinterestedin.Forexample,ifyouhadexecutedthesetcommandslistedpreviously,thenaskingforthewebapphashisassimpleastypingwebapp,asfollows:

chef:attributes>webapp

=>{"path"=>"/opt/webapp","db"=>{"username"=>"dbuser",

"password"=>"topsecret"},"user"=>"webuser"}

However,ifyouwishtoaccessthesewhenintherecipemode,theyareaccessedthroughthenodehash,asshownhere:

chef:attributes>recipe_mode

=>:attributes

chef:recipe>node[:webapp]

=>{"path"=>"/opt/webapp","db"=>{"username"=>"dbuser",

"password"=>"topsecret"},"user"=>"webuser"}

Theycanbeusedviathenodehashinjustthesamewayyouwouldusetheminarecipe.Ifyouwanttoconstructafileblockthatcreatedafoo.txtfilelocatedintheinstallpathofourwebapphash,youcaneasilyusethefollowingexampleinsideyourshell:

file"#{node[:webapp][:path]}/foo.txt"

Thismakeswritingrecipesusingtheinteractiveshellfeelexactlythesameaswritingrecipefiles.

UsingconfigurationblocksAresourceinarecipefilecanhaveaRubyblockwithattributes,andyoucandothisin

chef-shellinexactlythesamefashion.Simplyinsertdoaftertheresourcenameandtheshellwillbehaveasamultilineeditor,allowingyoutocompletetheblock.Thefollowingexampledemonstratesprovidingacontentattributetoafileresourceinthismanner:

chef:recipe>file"not_empty.txt"do

chef:recipe>content"Notempty!"

chef:recipe?>end

=><file[not_empty.txt]@name:"not_empty.txt"@noop:nil@before:nil

@params:{}@provider:Chef::Provider::File@allowed_actions:[:nothing,

:create,:delete,:touch,:create_if_missing]@action:"create"@updated:

false@updated_by_last_action:false@supports:{}@ignore_failure:false

@retries:0@retry_delay:2@source_line:"(irb#1):2:in'irb_binding'"

@guard_interpreter::default@elapsed_time:0@resource_name::file@path:

"not_empty.txt"@backup:5@atomic_update:true@force_unlink:false

@manage_symlink_source:nil@diff:nil@sensitive:false@cookbook_name:

nil@recipe_name:nil@content:"Notempty!">

Notethatwhentheshellprintedoutthepreviousfileresource,@contentwasnotpresent.Here,everythingbutthenameremainsthesame,andthereisanadditionalpropertyinsidetheobject,@content,asspecifiedinourattributesblock.

InteractivelyexecutingrecipesRunningarecipestepbystepisagoodwayofslowingdowntheexecutionofarecipesothatthestateofthesystemcanbeinspectedbeforeproceedingwiththenextresource.Thiscanbeincrediblyusefulbothfordebugging(aswillbediscussedlater)andfordevelopingandexploringresources.Itgivesyouachancetoseewhathashappenedandwhatsideeffectsyourrecipehasastherecipeisexecuted.Toachievethis,theChefshellallowsyoutorewindyourrecipetothestartandrunfromthebeginning,executeyourrecipeonestepatatime,andresumeexecutionfromthecurrentpointtotheend.

RestartingourChefshell,let’stakealookathowwecanusethis:

recipe_mode

echooff

file"foo.txt"

file"foo.txt"do

action:delete

file"foo.txt"do

content"Foocontent"

Hereourrecipeisquitesimple—createanemptyfile,foo.txt,removeit,andthenrecreateitwith“Foocontent”.Ifweexecuteourrecipeusingrun_chef,theshellwillperformalltheoperationsinonepasswithoutstoppingandwillnotallowustocheckwhetherthedeleteactionoccurred.Instead,wecanrunourrecipeandthenrewindandusethechef_run.stepmethodtointeractivelywalkthroughourrecipe:

chef:recipe>run_chef

...executionoutput…

chef:recipe>echoon

=>true

chef:recipe>chef_run.rewind

chef:recipe>chef_run.step

INFO:Processingfile[foo.txt]actiondelete((irb#1)line4)

INFO:file[foo.txt]backedupto/var/chef/backup/foo.txt.chef-

20140615175124.279917

file[foo.txt]deletedfileatfoo.txt

INFO:file[foo.txt]createdfilefoo.txt

INFO:file[foo.txt]updatedfilecontentsfoo.txt

INFO:file[foo.txt]ownerchangedto501

INFO:file[foo.txt]groupchangedto20

INFO:file[foo.txt]modechangedto644

Asyoucansee,herewewereabletorewindourrecipebacktothefirstinstruction(position0,astheresultofchef_run.rewindindicates),andthenwalkthrougheachresourcestepbystepusingchef_run.stepandseewhathappened.Duringthisrun,youcaneasilyopenaterminalafteryourewindtherecipe,deletethefoo.txtfilefromthepreviousrun,andcheckthatinitiallythereisnofoo.txtfile,thenstepthroughthenextcommandintherecipe,validatethatthereisanemptyfoo.txtfile,andsoon.Thisisaverygoodwaytolearnhowresourcesworkandtoseewhattheydowithouthavingtoformalizeyourrecipeinacookbook,provisionandbootstrapahost,andsoon.

DebuggingwiththeChefshellDebuggingisachievedintwodifferentwaysusingchef-shell:steppinginteractivelythrougharecipeorusingaspecialbreakpointresourcethatisonlyusedbychef-shell.Runningrecipesinteractivelystepbystepisgoodtobuildrecipeslocally;experimentwithresourcestodeterminetheeffectofcertainattributes,actionsandnotifications;ortoinspectthestateofthesystemaftereachresourcehasbeenactedupon.Thebreakpointsallowyoutoinjectveryspecificstoppingpointsintotheclientrunsothattheworldcanbeinspectedbeforecontinuing.Typically,onceabreakpointisencountered,youwillwanttostepthroughtheexecutionofyourscript(atleastforawhile)sothatthesearenotmutuallyexclusivetechniques.

UsingthebreakpointresourceThebreakpointresourceisstructuredjustlikeanyotherChefresource.Theresource’snameattributeisthelocationwhereyouwanttoinsertthebreakpoint,andithasonlyoneaction,:break,whichsignalschef-shelltointerruptexecutionofthecurrentrecipeandprovideaninteractiveshell.Anybreakpointresourcesinrecipesareignoredbythechef-client.Thatway,iftheyareforgottenaboutandleftinarecipe,theywillnotcausehavocinproduction.Thatbeingsaid,theyshouldonlybeusedwhenactivelydebugginganissueandremovedbeforereleasingyourrecipesintoyourproductionenvironment.

Thenameattributehasthefollowingstructure:

whenresourceresource_name

Here,whenhasthevalueof“before”or“after”,toindicatewhetherthebreakpointshouldstopbeforeorafterexecution,respectivelyandresourceisthetypeofresourcethatwhencombinedwithresource_nameistheuniqueidentifierthatwilltriggerthebreakpoint.Forexample:

beforefile'/tmp/foo.txt'

Thiswouldcausetheshelltointerruptexecutionoftherecipesimmediatelybeforeanyfileresourcethatwasmanipulating/tmp/foo.txt.Anotherexample,wherewewanttostopexecutionafterinstallingthegitpackage,wouldlooklikethefollowing:

afterpackage'git'

Usingthis,wewilltellchef-shellthatexecutionwastobepausedoncethegitpackagewasmodified.Let’slookathowwecanformasimplerecipecompletewithbreakpointresourcesthatwouldusetheseexamples:

breakpoint"beforefile'/tmp/foo.txt'"do

action:break

breakpoint"afterpackage'git'"do

action:break

file'/tmp/foo.txt'do

action:create

package'git'do

action:remove

Forthosewhohaveusedgdboranyotherdebugger,thiswillbeeasytounderstand;ifyouhavenotusedaninteractivedebugger,thentryafewoftheinteractiveexamples,andyouwillgetthehangofitinnotimeatall.

Chefshellprovidesacomprehensivewaytointeractwithyourrecipes.Nowthatyouseehowtotestoutanddebugyourwork,let’stakealookathowwecangoonestepfurtherinourtestingtoperformfullend-to-endintegrationtestingofourinfrastructure.

IntegrationtestingWithintegrationtesting,yourtestsmoveintotestingbeyondyourcode.WithChefSpec,welookedathowtoverifythatarecipewasexecutingstatementsasexpected.However,unittestinghasitslimits;integrationtestingcompletesthepicturebytestingcookbooksinconjunctionwithrealhoststhatrunthedesiredoperatingsystem(s):

Thismeanstestingcookbooksfromtheoutsideratherthanfromtheinside,whereaunittestprovidesverynarrowtestingofcodeinsideofcookbooks,integrationtestsperformdeploymentofyourcookbooksinatest-specificenvironmentandthenexecuteprobestovalidatethatthesystemisinthedesiredstate.OneofthemostpopulartoolsforthisisTestKitchen,whichwewilltakeabrieftourof.

UsingTestKitchenTestKitchenisanopensourcetoolthathelpstoautomateintegrationtestingofChefcookbooks.WhereaChefSpectestwouldvalidatethatafileresourcewascalledduringanexecution,asimilarlytaskedTestKitchentestwouldspinupanewinstance,executeyourrecipe(s),andthenvalidatethatthefilewasactuallycreatedonallplatformsthatyouexpecttherecipetoworkon.OneofthegreatthingsaboutTestKitchenisthatitsupportsanumberofdifferentdriverpluginstomanageyourtargethosts,includingVagrant(thedefault),EC2,OpenStack,Docker,andRackspaceCloudamongothers.ThisenablesyoutotestyourcookbooksnotonlyonlocalvirtualmachinesusingVagrant,buttoverifythattheyworkcorrectlyonacloudserviceaswell.Theabilitytoperformintegrationtestsondifferenttypesofhostsbringsyourteststhatmuchclosertomatchingaproductionenvironment,therebyincreasingyourconfidenceinthechangesyouaremaking.

InstallingTestKitchenCurrently,TestKitchenisdistributedasaRubygem,soinstallationisquitestraightforward:

geminstalltest-kitchen

Byinstallingthisgem,youarealsoinstallingacommand-linetool,kitchen,whichisusedtointeractwithTestKitchen.SimilartoVagrant,Bundler,andothertools,TestKitchenusesaconfigurationfiletostoreinformationaboutwhattotest,wheretotestit(onwhatvirtualmachines),andhowtotestit.

TestingwithTestKitchenAsmentionedbefore,TestKitchenfocusesonintegrationtestingofyourChefcomponents.Thismeansthatitneedstobeabletoexecuteyourrecipesonahost(orasetofhosts)andtheninvokeasetofteststovalidatethattheexpectedbehavioroccurredontheendhost(s).TestKitchenisresponsibleforthefollowing:

ProvisioningacleanhostfortestingInstallingChefontothenewhostExecutingourrecipesonthehostValidatingthebehavioroftherecipesonthehost

So,let’sgetstartedwithsomeactualtesting!

Buildingasimplecookbook

InordertodemonstratehowtouseTestKitchen,wewillneedtowriteacookbookthatwecanrunonourtesthostsandwriteteststovalidatethebehavior.InordertofocusonhowtouseTestKitchen,wewilltakealookataverysimplecookbookthatjustcreatesafileonthehostfilesystemsothatitisguaranteedthatitwillworkonbothUbuntuandCentOSplatforms.

First,createaplacetodoyourwork:

mkdir-ptestfile-cookbook/recipes

Then,addaverysimplemetadata.rbinyourtestfile-cookbookdirectorywiththefollowingcontentsinside:

name"testfile"

version"1.0"

Oncethatiscomplete,addadefaultrecipe,recipes/default.rb,withthesamplerecipecodeasfollows:

content"Myawesomefile!"

Nowthatwehaveacomplete(albeitsimple)cookbook,let’stakealookathowtotestitusingTestKitchen.

Preparingyourcookbookforthekitchen

InordertostartusingTestKitchen,youwillneedtoprepareyourtestenvironment.TestKitchenreliesonaconfigurationfile,.kitchen.yml,totellitwhattodo.Youcangenerateitbyhand,oryoucouldusetheinitcommandaspartofthekitchentool:

kitcheninit

Thiswilldoafewthingsforyou:first,itwillcreatea.kitchen.ymlfilewithsomesanedefaultsinthecurrentdirectory.Then,itwillcreateatest/integration/defaultdirectoryforyourintegrationtests,andthenitwillinstalltheVagrantdriverforTestKitchensothatitcaninteractwithVagrantvirtualmachines(ifithasnotalreadybeeninstalled).

Ifyoulookatthe.kitchen.ymlfile,youwillseethattheinitialfilecontainsthefollowingYAMLcode:

driver:

name:vagrant

provisioner:

name:chef_solo

platforms:

-name:ubuntu-12.04

-name:centos-6.4

suites:

-name:default

run_list:

-recipe[testfile::default]

attributes:

ThisconfigurationfileinstructsTestKitchentouseVagranttomanagethetargetinstancesandtouseChef-soloforprovisioning,anditshouldexecutethedefaultsuiteoftestsonbothUbuntu12.04andCentOS6.4.Ofcourse,youcanalwaysmodifyorextendthislistifyouhaveothersystemsthatyouwanttoruntestson,butthisisareasonabledefaultlist

fornow.

Noticethatwedon’thaveanyattributesspecifiedasourdefaultrecipeanditdoesnotuseattributes.Ifyouneedtoprovideattributestotestrecipes,thiswouldbetheplacetodoit,whichislaidoutasadictionaryinYAML.Eachtestsuitehasitsownrunlistanddefinedattributesthatallowyoutocheckthebehaviorofavarietyofconfigurationdataandrecipecombinations.

Testingyournewcookbook

TestingacookbookwithTestKitchenisoutlinedinthefollowingthreesteps:

1. Provisioningahostifneeded.2. Convergingthehostsothatitisuptodate.3. Executingtests.

Let’stakealookathowwewillperformthesewithoursimplecookbooktocheckthatitworksproperly.Provisioningtheinstance

Beforeyoucantest,youwillneedtopreparetheinstancesfortesting—thisisdoneusingthekitchencreate<instancename>command.Onlyyoudon’tknowwhatinstancetobringupjustyet.Togetthelistofinstancesthatcanberun,wewillusethelistsubcommand:

[jewart]$kitchenlist

InstanceDriverProvisionerLastAction

default-ubuntu-1204VagrantChefSolo<NotCreated>

default-centos-64VagrantChefSolo<NotCreated>

Youwillseethatthislistisgeneratedbyacombinationoftheplatformsandthesuiteslistedinthe.kitchen.ymlfile.Ifyouweretodefineanewsuite,namedserver,thenyourlistwouldincludetwoadditionalinstances,server-ubuntu-1204andserver-centos-64.

Onceyouhaveseenthislist,youcancreateanUbuntu12.04instancewiththefollowingcommand:

kitchencreatedefault-ubuntu-1204

ThiswilluseVagrantandVirtualBoxtoprovisionanewheadlessUbuntu12.04hostandbootitupforyoutostarttestingwith.Ifyoudon’talreadyhaveanUbuntu12.04Vagrantimagedownloaded,thenitwillbedownloadedforyouautomatically(thisisalargeimagesoitmaytakeawhiletocompletethisoperation,dependingonyourconnectionspeed,ifyouarefollowingalong).

ThiswilllookfamiliartothosethatusedVagrantatthebeginningofthechapter:

[jewart]%kitchencreatedefault-ubuntu-1204

----->StartingKitchen(v1.2.1)

----->Creating<default-ubuntu-1204>...

Provisioninghappens…

Finishedcreating<default-ubuntu-1204>(4m2.59s).

----->Kitchenisfinished.(4m2.83s)

However,thisonlybuildsthevirtualmachineforourtests;itdoesnotrunourrecipesorourtestsinthenewlyconstructedhost.Beforewemoveontowritingatest,let’stakealookathowtorunourrecipeinsideourinstance.Convergingthenewlycreatedinstance

Thenextstepistoexecuteourrunlist(“converge”inChefparlance)onthenewinstance.ThisisdonewithTestKitchen’sconvergecommandandcanbeusedforalloronespecificinstance.InordertoconvergeourUbuntu12.04instance,thefollowingcommandisused:

kitchenconvergedefault-ubuntu-1204

Whatthiswilldoistransferanyrequireddatafiles,installChefasneeded,andthenexecutetherunlistforthespecifiedsuite(defaultinthiscase)ontheinstance.Hereisasamplerunofconverge(withsomepartsremoved):

[jewart]%kitchenconvergedefault-ubuntu-1204

----->Converging<default-ubuntu-1204>...

Preparingfilesfortransfer

Preparingcurrentprojectdirectoryasacookbook

Removingnon-cookbookfilesbeforetransfer

----->InstallingChefOmnibus(true)

InstallationofChefhappens…

CompilingCookbooks…

Converging1resources

Recipe:testfile::default

AndthenyouwillseetheoutputofanormalChefrunafterthat—atthispoint,therunlistiscompleteandtheinstancehasbeenconvergedtothelateststate.Nowthatit’sbeenconverged,wewillcontinuetowriteaverysimpletesttoverifythatourrecipedidtherightthing.Writingasimpletest

TestsarestoredinadirectorywhosestructureissimilartootherChefcomponents—thedirectorywecreatedpreviouslywithkitcheninit,tests/integration/default,allowsustokeepourintegrationtestsseparatefromspectestsorothertypesoftests.Theintegrationdirectorywillcontainonedirectorypersuitesothattestfilesaregroupedtogetherbasedontheparticularcomponentoraspectofyourcookbookthatisbeingtested.Additionally,dependingonthetypeofthetestframeworkbeingused,yourtestswillbecontainedinanotherchilddirectoryforthegivensuite.Inthiscase,wewilltakealookattheBASHAutomatedTestingSystem(BATS),soourtestwillbeplacedinthetests/integration/default/bats/file_created.batsfileandwilllooklikethefollowingcode:

#!/usr/bin/envbats

@test"myfile.txtexistsin/tmp"{

[-f"/tmp/myfile.txt"]

Thisallowsustousethesimple-fBASHtest(whichreturnstrueifthespecifiedvalueexists)toguaranteethatthefilewascreatedontheinstance.

Next,wecanrunthistestwithkitchenverifydefault-ubuntu-1204andseethattheBATSpluginwasinstalledandthatourtestwasexecutedandpassed:

[jewart]%kitchenverifydefault-ubuntu-1204

----->Settingup<default-ubuntu-1204>...

Fetching:thor-0.19.0.gem(100%)

Fetching:busser-0.6.2.gem(100%)

Successfullyinstalledthor-0.19.0

Successfullyinstalledbusser-0.6.2

2gemsinstalled

----->SettingupBusser

CreatingBUSSER_ROOTin/tmp/busser

Creatingbusserbinstub

Pluginbatsinstalled(version0.2.0)

----->Runningpostinstallforbatsplugin

InstalledBatsto/tmp/busser/vendor/bats/bin/bats

Finishedsettingup<default-ubuntu-1204>(0m8.94s).

----->Verifying<default-ubuntu-1204>...

Suitepathdirectory/tmp/busser/suitesdoesnotexist,skipping.

Uploading/tmp/busser/suites/bats/file_created.bats(mode=0644)

----->Runningbatstestsuite

√myfile.txtexistsin/tmp

1test,0failures

Finishedverifying<default-ubuntu-1204>(0m0.67s).

----->Kitchenisfinished.(0m9.86s)

Todemonstratewhathappensifthefiledoesnotexist,wecancloneourtesttocreateasecondsimpletestthatvalidatestheexistenceof/tmp/myotherfile.txt,andrunourverifycommandagainwithoutmakingacorrespondingchangetoourrecipe.TheoutputfromTestKitchenwilltellusthatourtestfailedandwhy:

----->Runningbatstestsuite

√myfile.txtexistsin/tmp

myotherfile.txtexistsin/tmp

(intestfile/tmp/busser/suites/bats/other_file.bats,line4)

2tests,1failure

Command[/tmp/busser/vendor/bats/bin/bats/tmp/busser/suites/bats]exit

codewas1

>>>>>>Verifyfailedoninstance<default-ubuntu-1204>.

Combiningallthesteps

Fortunately,thefinefolksthatcreatedTestKitchenrealizedthatitwouldbetedioustorunallthreestepseverytimeyouwantedtorunsometests.Asaresult,thereisthekitchentestcommandthatwillprovisionaninstance,executetherunlist,verifytheresults,andthenteardowntheinstancewithonlyonecommand.Inthiscase,youcanreplacethemwiththefollowingsinglecommand:

kitchentestdefault-ubuntu-1204

ThiscoversthebasicsoftestingyourcookbookswithTestKitchen.ThereareotherthingsthatcanbedonewithTestKitchen,includingusingothertestingmechanisms,testingcookbookdependencies,validatingwhetherservicesarerunning,fullyautomatingTestKitchenaspartofyourreleaseprocess,andplentymore.Formoreinformation,visittheprojectathttp://kitchen.ci/.

ExtendingChefChefisdevelopedintheopenwithflexibilityandextensibilityinmind.Mostofthetoolsarearchitectedtosupportloadingcustompluginstosupportdevelopmentofadd-onsfornewfunctionality.Asyousawearlier,Knife’scloudservicesupportisprovidedbyplugins,oneforeachcloudservice,includingEC2,Azure,andRackspacecloud.WewilllookathowthathappenssothatyoucanexplorewritingyourownpluginsforKnife,Ohai,andotherChefcomponentsshouldtheneedarise.

InadditiontoextendingChef’scorecomponentsdirectly,itispossibletoextendthefunctionalityofyourChefecosystembybuildingenhancementstoexistingtoolsthatcanleverageChef’sdataAPIstoprovidedataaboutyourinfrastructure.

WritinganOhaipluginOhai(aplayonthephraseoh,hi!)isatoolthatisusedtodetectattributesonanode,andprovidetheseattributestothechef-clientatthestartofeverychef-clientrun.WithoutOhai,thechef-clientwillnotfunction,andtherefore,itmustbepresentonanodeinorderforCheftowork.Thedatathatiscollectedisauthoritative—ithasthehighestlevelofprecedencewhencomputingattributedataforclientrunsonanode.

ThetypesofattributesthatOhaimightbeusedforinclude:

PlatformdetailsNetworkusageMemoryusageProcessorusageKerneldataHostnamesInformationaboutthenetworktopologyCloud-specificinformation

Ohaiimplementsanextensiblearchitecturethroughpluginsthatallowsenduserstowritecustomextensionstoreportinformationthatiscollectedaboutanode.Forexample,therearepluginsforEC2cloudhoststhatuseEC2-specificmechanismstodetermineinformationaboutthehost,includingitsinternalIPaddressandotherbitsofinformation.

ThisisincrediblyusefulforintegratingChefwithanexistinginfrastructure,asyoucanautomaticallyprobeforlocalconfigurationdataandgenerateattributesfromthatdata.OncethisdataisstoredinChef,itcanbeusedinsearchqueries,recipes,andeverywhereelsethatyoucouldotherwiseusenodeattributes.

EveryOhaipluginfollowsapattern:itregistersitselfasprovidingacertainclassofattributedataandcontainsacombinationofgeneralpurposeanddatacollectionmethodstogatherinformationaboutthelocalstateofthesystemandreportitbacktotheChefserver.Ohaialreadyhasbuilt-incollectorsforanumberofplatforms,includingLinux,Windows,andBSD.

AsampleOhaipluginwouldlooklikethefollowing:

Ohai.plugin(:Region)do

provides"region"

definit

regionMash.new

collect_data(:default)do

#Runsonallhostswhoseplatformisnotspecificallyhandled

region[:name]="unknown"

collect_data(:linux,:freebsd)do

#RunonlyonLinuxandFreeBSDhosts

region[:name]=discover_region_unix

collect_data(:windows)do

#RunonWindowshosts

region[:public_ip]=discover_region_windows

Hereweregisterourpluginasprovidingaregiondata,thereisaninitmethodthatcreatesaMashforourregiondata,andafewdatacollectioncallbacks.Eachdatacollectioncallbackisregisteredasablockthatiscalledforthespecifiedplatform(s).Inourcase,therearethreecallbacksregistered,oneforWindowssystems,oneforLinuxandFreeBSDsystems,andthenafallbackthatwillbecalledforanyplatformnotexplicitlyhandled.

AnoteaboutwritingOhaipluginsThewaywedeclaretheregionMashinourinitmethodisalittlebitdifferentthannormalvariableassignmentinRuby.Inourplugin,wedefinetheplugin’sregionpropertywiththefollowingcode:

regionMash.new

Onthesurface,thismightlooklikesomeoneforgotanequalssign,asinthefollowingRubycode:

region=Mash.new

However,iftherewereanequalssign,thepluginwouldnotworkasintended.Inthiscase,thereisnoequalssignmissingandthecodeiscorrect.ThisisbecauseOhai’sPluginclassleveragesaspecialRubymechanismforinterceptingcallstononexistentmethodsanddynamicallyhandlingthem.Thismechanismisrecognizablebythepresenceofaspecialmethodnamedmethod_missinginthecode.Inthiscase,themethod_missinghandlerwillcallaspecialOhaipluginmethod,get_attribute,ifnoargumentsarepassed,oritwillcalltheset_attributemethodifargumentsarepassed.

Todemonstratewhythisisused,ifyouwantedtohavethesameeffectwithoutthemethod_missingmechanism,thentheplugin’sinitmethodcouldbewrittenas:

definit

set_attribute"region",Mash.new

Wereyoutodothis,thenoursubsequentcollectdatamethodswouldneedtoberewrittenaswell.Hereisanexampleofwhattheymightlooklike:

collect_data(:platform)do

region=get_attribute"region"

region[:name]=get_data_mechanism

set_attribute"region",region

Youcanseethatthemethod_missingmechanismmakeswritingpluginsmorenatural,thoughittakesalittlebitofextraworktounderstandhowtowritethematfirst.

ChefwithCapistranoOneexampleofextendingChefthroughexternaltoolsisthecapistrano-chefgemthatextendsthepopulardeploymenttool,Capistrano.WritteninRuby,Capistranowasdesignedtodeployapplicationsandperformlightsystemsadministration.IfyouhaveexistingapplicationsthatarebeingdeployedusingCapistrano,thisisanexampleofhowtoleverageyourconfigurationdatastoredinCheftomakeintegrationasseamlessaspossible.

IfyouhaveanexistingapplicationthatusesCapistrano,youwillhaveadeploy.rbfilethatdefinesthevariousapplicationroles.EachtoolhasanarrayofIPaddressestohoststhatprovidethatroleandmightlooksomethinglikethis:

role:web,'10.0.0.2','10.0.0.3'

role:db,'10.0.0.2',:primary=>true

Byusingcapistrano-chef,youcandothis:

require'capistrano/chef'

chef_role:web,'roles:web'

chef_role:db,'roles:database_master',

:primary=>true,

:attribute=>:private_ip,

:limit=>1

NoticethatherewehaveusedasimplesearchquerytodeterminethehoststhatshouldbeincludedineachCapistranorole.Inthiscase,the:webrolehasbeenreplacedwithaChefsearchqueryforallnodesthathavethewebChefroleassociatedwiththem.ThisallowsyoutomodelyourdatainChef,butstilluseCapistranotodeployyourapplicationstack,increasingtheeaseofintegration.

ThisworksbecauseallofChef’sdataisavailableviaanHTTPAPImakingintegrationassimpleasmakingHTTPcallsandparsingsomeJSONresults(which,comparedtosomeotherintegrationmechanisms,isincrediblyeasy).

AutomationandintegrationOneofthebestpartsaboutChefisthatyourinfrastructureandsoftwaremodelisconsistentwithwhatisdeployed.Whatthismeansforyouisthatwhenchef-clientrunsonanendhost,thathost’sstateisupdatedtomatchyourmodeledenvironment.Forexample,considerascenariowhereyouhave10EC2databasehosts,andallofthemhaveaspecialrole,database_server,appliedtothem.Thisrole’sattributesindicatethatPostgreSQL9.1istobeinstalledanditsdatashouldbestoredin/opt/postgresql/data.Byexecutingchef-clientonalltennodes,theywillhavePostgreSQL9.1installedandstoringdatain/opt/postgresql/data.NowconsiderthatallofournodesneedtohaveanewEBSstoragedeviceattachedtoeachofthem,andPostgreSQLneedstobepointedtoournewEBSdevice.UpdatingourmodeltoincludearecipethatmountstheEBSdevicegracefullyshutsdownPostgreSQL,movesthedatabasedata,reconfiguresPostgreSQL,andstartsitupagain.Wecanautomateandrolloutthisconfigurationtoourfleetoftendatabasehosts.Youcaneasilyimaginetenhostsgrowingtohundredsoreventhousands.Thisiswhatthepowerofautomationisallabout.

AutomatedupdatesanddeploymentsIfyouhaveconfidenceinyourmodelandyourcookbooks,thenyoucantakethisautomationonestepfurther.Byautomatingtheexecutionofchef-clientonaperiodicschedule,youcanfullyautomateupdateswithoutneedingtoSSHintothehoststorunchef-client.However,thislevelofautomationrequiresahighlevelofconfidenceinthecorrectnessofyourcookbooks.Achievingthisrequirescontinuousandin-depthtestingofnotonlythecodeinthecookbooksdevelopedbutalsoofthedependenciesthatareneededtomakeyourcookbookswork.Tothatend,comprehensiveintegrationtestscanhelptobuildtheconfidenceneededtomoveintoafullyautomatedworld.

SummaryBynowyouhavebeenexposedtoalotofwhatChefhastooffertheDevOpscommunity.YouhaveseenwhatChefdoes,howtoinstallit,andhowitworks.Throughoutthisbook,youhavebeenintroducedtosomenewwaysofthinkingabouthowtomodelinfrastructureanduseautomatedtoolstomanageit.

Atthispoint,youhopefullyunderstandhowtomodelyourinfrastructurewithChefaswellasinstallthevariouscomponentsrelatedtoChef,rangingfromtheservertotheclient.Fromhere,youcantakewhatyou’velearnedaboutthevariouscomponentsofChefandusethatinformationtobuildmoreadvancedcookbookstodeployyoursoftwareandmanageyourinfrastructure,rangingfromcloudhoststophysicalon-sitehardwareandevenvirtualmachinesusingVagrant.Onceyouhavegottenthingsworking,youcanautomateyourconfigurationtoolsandensurethereliabilityofyourcookbooksthroughunitandintegrationtestsaswell.

Now,itisyourturntotakethisinformationandyournewskillstoautomateyoursystemsinfrastructureinordertobuildexcitingnewthings!

IndexA

AmazonEC2about/AmazonEC2workingwith/AmazonEC2knifeplugin,installing/InstallingtheEC2knifepluginauthentication,settingup/SettingupEC2authenticationinstance,provisioning/Provisioninganinstanceinstance,bootstrapping/Bootstrappingtheinstanceinstance,terminating/TerminatingtheinstanceChefnode,removing/RemovingtheChefnode

AmazonEC2authenticationsettingup/SettingupEC2authentication

AMIIDURL/Provisioninganinstance

ApacheSolrdocumentationURL/Queryingyourdata

applicationconfiguring/Configuringyourapplication

applicationdeploymentcookbookviewing/Lookingatyourapplicationdeploymentcookbook

applicationrunmaintaining/Keepingyourapplicationrunning

applicationtemplatedefining/Definingafullapplicationtemplate

attributes/GettingtoknowChefabout/Attributesconfigurationvalues/Attributesmultiplefiles/Multipleattributefilesusing/Usingattributes

authorizedkeys,SSHkeystemplating/Templatingtheauthorizedkeys

automation,Chef/Automationandintegration

BBase64encoding/Secretkeysbaselinerole/ModelingasimplePythonapplicationbaseserverrole

creating/CreatingthebaseserverroleBASHAutomatedTestingSystem(BATS)/Writingasimpletestbootstrap/GettingtoknowChefbootstrapscript/GettingtoknowChefbreakpointresource,Chefshell

using/UsingthebreakpointresourceBundler,ChefSpec/InstallingChefSpec

CCapistran

Chef,usingwith/ChefwithCapistranoChef

terminology/Terminologyworkingwith/WorkingwithChefbenefits/WorkingwithChef,Large-scaleinfrastructurechef-solo,installing/Installingchef-soloRubygemmechanism/TheRubygemgems,managing/Managinggemschef-solo,verifying/Verifyingthatchef-soloworksabout/GettingtoknowChefnode/GettingtoknowChefworkstation/GettingtoknowChefbootstrap/GettingtoknowChefbootstrapscript/GettingtoknowChefrecipe/GettingtoknowChefcookbook/GettingtoknowChefattributes/GettingtoknowChefrole/GettingtoknowChefrunList/GettingtoknowChefresource/GettingtoknowChefprovider/GettingtoknowChefdatabags/GettingtoknowChefenvironment/GettingtoknowChefused,forsoftwaredeployment/DeployingsoftwarewithChefdependencies,managing/ManagingdependenciesinChefextending/ExtendingChefOhaiplugin,writing/WritinganOhaipluginusing,withCapistran/ChefwithCapistranoautomation/Automationandintegrationintegration/Automationandintegrationupdates,automating/Automatedupdatesanddeployments

chef-server-ctl,Chefserverworking/Understandinghowchef-server-ctlworks

Chef-soloabout/VagrantandChef-soloVagrant,combiningwith/CombiningVagrantwithChef-solo,ConfiguringChef-solo,TellingChef-solowhattorunlimitations/UnderstandingthelimitationsofChef-soloconfiguring/ConfiguringChef-solocookbooks_path/ConfiguringChef-solodata_bags_path/ConfiguringChef-solo

roles_path/ConfiguringChef-solorundirections,providing/TellingChef-solowhattorunroles,usingwith/UsingrolesanddatabagswithChef-solodatabags,usingwith/UsingrolesanddatabagswithChef-solo

chef-soloinstalling/Installingchef-soloverifying/Verifyingthatchef-soloworks

ChefcommunityURL/Managingthecookbooks

ChefLWRP,componentresource/Buildingaresourceprovider/Buildingaresource

Chefnode,AmazonEC2removing/RemovingtheChefnode

Chefnode,RackspaceCloudremoving/RemovingtheChefnode

Chefserverinstalling/InstallingaChefserverURL/Gettingtheinstallerconfiguring/ConfiguringaChefserverinteracting,Chefshellused/InteractingwiththeChefserverusingtheshell

Chefserver,installingonRedHatEnterpriseLinuxpackage,downloading/Downloadingthepackagepackage,downloading,URL/Downloadingthepackage

Chefserver,installingonUbuntuabout/InstallingonUbuntupackage,downloading/Downloadingthepackagepackage,downloading,URL/Downloadingthepackagepackage,installing/Installingthepackage

Chefserverconfigurationabout/ConfiguringaChefserverchef-server-ctl,working/Understandinghowchef-server-ctlworkshostserveractions,monitoring/What’shappeningonmyserver?services,verifying/Verifyingthattheservicesarerunning

Chefserverinteraction,withChefshelldata,interactingwith/Interactingwithdatadata,searching/Searchingyourdatadata,editing/Editingyourdatadata,transforming/Transformingdata

Chefserviceabout/Terminologyvalidating/Validatingthatyourserviceisworkingknifeconfiguration/Ensuringthatyourknifeconfigurationworks

Chefshell

about/GettingtoknowtheChefshellusing/UsingtheChefshellstandalonemode/Thestandalonemodesolomode/Thesolomodeclientmode/Theclientmodeused,forinteractingwithChefserver/InteractingwiththeChefserverusingtheshellused,forexecutingrecipes/ExecutingrecipeswithChefshellrecipe,creatingin/Creatingarecipeintheshellused,fordebugging/DebuggingwiththeChefshellbreakpointresource,using/Usingthebreakpointresource

ChefSpecabout/TestingrecipesandRSpec/RSpecandChefSpectestingbasics/Testingbasicsusing/UsingChefSpecoverview/GettingstartedwithChefSpecinstalling/InstallingChefSpecdependencies,lockinginRuby/LockingyourdependenciesinRubysimplerecipe,creating/CreatingasimplerecipeandamatchingChefSpectest

ChefSpectestwriting/WritingaChefSpectestexpanding/Expandingyourtestsmultipleexamples/Multipleexamplesinaspectesttesting,formultipleplatforms/Testingformultipleplatforms

ChefSupermarketURL/Determiningwhichrecipesyouneed

clientmode,Chefshell/Theclientmodecloud

leveraging/Leveragingthecloudcloudplatformproviders

AmazonEC2/AmazonEC2RackspaceCloud/RackspaceCloud

components,forChefserverinstallation/Whatyouwillbeinstallingconfigurationblocks

using/Usingconfigurationblocksconfigurationdata

organizing/Organizingyourconfigurationdatasources/Organizingyourconfigurationdataattributedataexample/Exampleattributedatadatabags/Databags

cookbook/Terminologyabout/GettingtoknowChef,Installingacookbookinstalling/Installingacookbook

cookbook,TestKitcheninstance,provisioning/Provisioningtheinstancecreatedinstance,converging/Convergingthenewlycreatedinstancetest,writing/Writingasimpletestsummarizing/Combiningallthesteps

cookbooksmanaging/ManagingthecookbooksURL/Managingthecookbooksdownloading/DownloadingcookbooksURL,fordownloading/Downloadingcookbooksdatabaserecipe,viewing/Lookingatthedatabaserecipeapplicationdeploymentcookbook,viewing/Lookingatyourapplicationdeploymentcookbookdirectories,preparing/PreparingthedirectoriesPythonvirtualenvironment,constructing/ConstructingyourPythonvirtualenvironmentsourcecode,checkingout/Checkingthesourcecodedependencies,installing/InstallinganyextradependenciesPython’srequirementsfile,using/UsingPython’srequirementsfileapplication,configuring/Configuringyourapplicationapplicationrun,maintaining/Keepingyourapplicationrunning

customdefinition,developingabout/Developingacustomdefinitioncode,organizing/Organizingyourcodedefinition,writingforusingPIP/WritingadefinitionforusingPIPapplicationtemplate,defining/Definingafullapplicationtemplate

customextensionswriting/Writingcustomextensionsresource,building/Buildingaresource

storing,indatabags/Storingdataindatabagssearching/Searchingfordata,Searchingyourdataencrypting/Encryptingyourdatadecrypting/Decryptingyourdataquerying/Queryingyourdata

data,Chefserverinteracting/Interactingwithdatasearching/Searchingyourdataediting/Editingyourdatatransforming/Transformingdata

data,searchingdatabags,searchingwithknifetool/Searchingyourdatabagswithknifedatabags,searchingfromrecipes/Searchingyourdatabagsfromarecipedata,querying/Queryingyourdata

databagsabout/GettingtoknowChef,Databagsusing/Knowingwhentousedatabagsdata,storingin/Storingdataindatabagscreating,forusers/Creatingadatabagforusersworkingwith/Workingwithdatabagssecuring/Securingyourdatabagsdata,searching/Searchingyourdatabagswithknifemultiplemachines,managingwithsearchqueries/Managingmultiplemachineswithsearchqueries

databagssearching,withknifetool/Searchingyourdatabagswithknifesearching,fromrecipes/Searchingyourdatabagsfromarecipe

databags,securingabout/Securingyourdatabagssecretkeys/Secretkeysdata,encrypting/Encryptingyourdatadata,decrypting/Decryptingyourdatakeys,storingonnodes/Storingkeysonnodes

databags,usingwithChef-soloabout/UsingrolesanddatabagswithChef-solocustomJSONdata,injecting/InjectingcustomJSONdatacustomnodename,providing/Providingacustomnodename

databasehostconfiguring/Configuringthedatabasehost

databaserecipeviewing/Lookingatthedatabaserecipe

databaseserverrolecreating/Creatingthedatabaseserverrole

defaultaction/Overridingadefaultbehaviordefaultbehavior,resource

overriding/Overridingadefaultbehaviordefinition/Developingacustomdefinitiondefinitions

about/Definitionsdependencies

installing/Installinganyextradependenciesmanaging,inChef/ManagingdependenciesinChefmanaging/Managingdependencieselsewhere

deploymentkeys,SSHkeysadding/Addingdeploymentkeys

deployusers/Addingdeploymentkeysdirectories

preparing/Preparingthedirectoriesdriverplugins/UsingTestKitchendry-runmode/Implementingtheprovider

EEC2instances

provisioning/ProvisioningEC2instancesdatabasehost,configuring/Configuringthedatabasehost

echooffstate/Creatingarecipeintheshellechoonstate/Creatingarecipeintheshellencryptionkey

storing,onnodes/Storingkeysonnodesenvironment

about/GettingtoknowChefexample/Environments

equalsign/VariablereplacementERBprimer

about/AquickERBprimerURL/AquickERBprimerRubycode,executing/ExecutingRubyRubycodeexecution,variablereplacement/Variablereplacement

Ffetch_from_s3method/Implementingtheprovider

managing,withRVM/Managinggemsgemset/Managinggems

HHostedChef

URL/WorkingwithChef

IImage-O-Rama/Determiningwhichrecipesyouneedimage-processingrole

defining/Animage-processingroleimagesearchrole

defining/Animagesearchroleinfrastructuremodeling

steps/Modelingyourinfrastructureabout/ModelingyourinfrastructureChef,using/Modelingyourinfrastructureservice-orientedwebapplication,services/Modelingyourinfrastructureservice-orientedarchitecture,advantages/Modelingyourinfrastructurerole/Rolesrole,implementing/Implementingarolerecipes,determining/Determiningwhichrecipesyouneedrecipes,applyingtorole/Applyingrecipestorolesrole,mappingtonode/Mappingyourrolestonodesrole,mappingtonodes/Mappingyourrolestonodesnodes,converging/Converginganodeenvironment/Environments

installationVagrant/InstallingVagrant

installation,Chefserverabout/InstallingaChefserverrequisites/Requirementsandrecentchanges,Installationrequirementscomponents/Whatyouwillbeinstallingomnibusinstaller,obtaining/GettingtheinstalleronUbuntu/InstallingonUbuntuonRedHatEnterpriseLinux/InstallingonRedHatEnterpriseLinux

instance,AmazonEC2provisioning/Provisioninganinstancebootstrapping/Bootstrappingtheinstanceterminating/Terminatingtheinstance

instance,RackspaceCloudprovisioning/Provisioninganinstanceterminating/Terminatinganinstance

integration,Chef/Automationandintegrationintegrationtesting

about/IntegrationtestingTestKitchen,using/UsingTestKitchen

Kknife-rackspaceplugin/Provisioninganinstanceknifenodelist/InteractingwiththeChefserverusingtheshellknifeplugin,AmazonEC2

installing/InstallingtheEC2knifepluginknifesearchnode…/InteractingwiththeChefserverusingtheshellknifetool/Validatingthatyourserviceisworking

used,forsearchingdatabags/Searchingyourdatabagswithknife

Llarge-scaleinfrastructure/Large-scaleinfrastructureload_current_resourcemethod/Loadinganexistingresourcelocalenvironment

configuring/Configuringyourlocalenvironment

MMash,Chef/UsingattributesmatchingChefSpectest

creating/CreatingasimplerecipeandamatchingChefSpectestmetadata

about/Metadatamockmethods/RSpecandChefSpecmultipleattributefiles

about/Multipleattributefilesmultipleplatforms,supporting/Supportingmultipleplatformsexternalattributes,loading/Loadingexternalattributes

multiplemachinesmanaging,withsearchqueries/Managingmultiplemachineswithsearchqueries

mysql/Lookingatthedatabaserecipe

N*nameparameter/WritingadefinitionforusingPIPnewhost

provisioning,withVagrant/ProvisioninganewhostwithVagrantnode/Terminology

about/GettingtoknowChefrole,mappingto/Mappingyourrolestonodesconverging/Converginganode

nodeattributesdefining/Definingnodeattributessetting/Settingattributesaccessing/Accessingattributes

nodedeletecommand/RemovingtheChefnodenodehash/Usingattributes

OOhaiplugin

writing/WritinganOhaiplugin,AnoteaboutwritingOhaipluginsattributes/WritinganOhaiplugin

omnibusinstallationpackage,Chef/Requirementsandrecentchangesomnibusinstaller,Chefserver

obtaining/Gettingtheinstalleroutline/Installationoutlinesteps/Installationoutline

OpenSSHserviceroledefining/AnOpenSSHservicerole

used,forwritingdefinition/WritingadefinitionforusingPIPpostgresql/LookingatthedatabaserecipePostgreSQLservicerole

defining/APostgreSQLserviceroleprovider/GettingtoknowChef

about/Buildingaresourceimplementing/Implementingtheprovider

Pythonapplicationmodelling/ModelingasimplePythonapplication

Pythonrequirementsfileusing/UsingPython’srequirementsfile

Pythonvirtualenvironmentconstructing/ConstructingyourPythonvirtualenvironment

Python’srequirementsfileusing/UsingPython’srequirementsfile

RRackspaceCloud

about/RackspaceCloudinstance,provisioning/Provisioninganinstanceinstance,terminating/TerminatinganinstanceChefnode,removing/RemovingtheChefnode

read-eval-print-loop(REPL)/GettingtoknowtheChefshellrecipe/Terminology

about/GettingtoknowChefdetermining/Determiningwhichrecipesyouneedneedfor/Determiningwhichrecipesyouneedcookbook,installing/Installingacookbookapplying,torole/Applyingrecipestorolesexecuting/Startingoutsmalltesting/Testingrecipestesting,withChefSpec/Testingrecipesbuilding/Buildingyourrecipetests,executing/Executingtestsfailures/Understandingfailurescreating,inChefshell/Creatingarecipeintheshell

recipesabout/Recipes,Recipesstartingstates/Recipesdeveloping/Developingrecipeswriting/Writingrecipessimpleservice,installing/Installingasimpleservicecomplicatedrecipe,actions/Gettingmoreadvancedenhancing,withsearchmethod/Searchinginsiderecipesdatabags,searchingfrom/Searchingyourdatabagsfromarecipe

recipes,executingwithChefshellabout/ExecutingrecipeswithChefshell,Interactivelyexecutingrecipesrecipemode/Creatingarecipeintheshellattributemode/Creatingarecipeintheshellnodeattributes,defining/Definingnodeattributesattributes,setting/Settingattributesattributes,accessing/Accessingattributesconfigurationblocks,using/Usingconfigurationblocks

RedHatEnterpriseLinuxChefserver,installingon/InstallingonRedHatEnterpriseLinux

resource/GettingtoknowChefbuilding/Buildingaresourcepackageresource/Buildingaresourcedefining/Definingtheresource

provider,implementing/Implementingtheprovidermodifying/Modifyingresourcesexistingresource,loading/Loadinganexistingresourceupdatedstatus,declaring/Declaringthataresourcewasupdated

resourcesabout/Resourcesbuilt-inresources/Resourcesusing/Usingresourcesdefaultbehavior,overriding/Overridingadefaultbehavior

reusableresources,defininginChefbenefits/Definingafullapplicationtemplate

roleabout/GettingtoknowChef,Rolesdefining/Definingrolesimplementing/Implementingaroleapplying,torecipes/Applyingrecipestorolesmapping,tonode/Mappingyourrolestonodes

role,definingabout/Definingroleswebapplicationservicerole/Awebapplicationserviceroleimage-processingrole/Animage-processingroleimagesearchrole/AnimagesearchrolePostgreSQLservicerole/APostgreSQLserviceroleSolrservicerole/ASolrserviceroleOpenSSHservicerole/AnOpenSSHservicerole

rolesdefining/Definingrolesbaseserverrole,creating/Creatingthebaseserverroledatabaseserverrole,creating/Creatingthedatabaseserverrolewebserverrole,creating/Creatingthewebserverrole

roles,usingwithChef-soloabout/UsingrolesanddatabagswithChef-solocustomJSONdata,injecting/InjectingcustomJSONdatacustomnodename,providing/Providingacustomnodename

RSpecabout/RSpecandChefSpec/RSpecandChefSpectestinglevels/RSpecandChefSpeccomparing,withtestinglibraries/ComparingRSpecwithothertestinglibrariesfailures/Understandingfailures

Rubydependencies,locking/LockingyourdependenciesinRuby

Rubygemmechanism/TheRubygemRubyVersionManager(RVM)

about/TheRubygemURL/Managinggems

runlist/GettingtoknowChefrunner/Testingformultipleplatforms

Ssearchmethod

used,forenhancingrecipes/Searchinginsiderecipessearchqueries

used,formanagingmultiplemachines/Managingmultiplemachineswithsearchqueries

secretkeys,databagsmanaging,withsearchqueries/Secretkeys

serverlistsubcommand/Terminatinganinstanceservice-orientedarchitecture(SOA)/VagrantandChef-solosetup

describing/Describingthesetupshelluserrecipe

evolution/Evolutionofashelluserrecipesoftware

deploying,withChef/DeployingsoftwarewithChefdeploying/Deployingyoursoftware

softwaredeploymentupdates,deployingmanually/Manuallydeployingupdatesautomating/Automatingdeployment

solomode,Chefshell/ThesolomodeSolrservicerole

defining/ASolrservicerolesourcecode

checking/CheckingthesourcecodeSSHkeys

distributing/DistributingSSHkeysauthorizedkeys,templating/Templatingtheauthorizedkeysdeploymentkeys,adding/Addingdeploymentkeys

standalonemode,Chefshell/Thestandalonemodesupervisordservice

URL/Keepingyourapplicationrunning

Ttemplates

about/Templatesneedfor/Whyusetemplates?ERBprimer/AquickERBprimertemplateresource/Thetemplateresourcevariables/Thetemplatevariablesvariables,passingto/Passingvariablestoatemplatecomputedconfiguration,accessing/Accessingcomputedconfigurationssearchingfor/Searchingfortemplatessearchorder/Searchingfortemplates

terminology,Chefnode/TerminologyChefservice/Terminologyworkstation/Terminologyrecipe/Terminologycookbook/Terminology

test-driven-development(TDD)/TestingbasicsTestKitchen

using/UsingTestKitcheninstalling/InstallingTestKitchenused,fortesting/TestingwithTestKitchentasks/TestingwithTestKitchencookbook,building/Buildingasimplecookbookcookbook,preparingfor/Preparingyourcookbookforthekitchencookbook,testing/Testingyournewcookbook

UUbuntu

Chefserver,installingon/InstallingonUbuntuupdated_by_last_actionmethod/Declaringthataresourcewasupdatedusercookbook

enhancing/Enhancingyourusercookbookusermanagement

Chef/Managingusersshelluserrecipe,evolution/Evolutionofashelluserrecipedata,storingindatabags/Storingdataindatabagsrecipe,enhancingwithsearchmethod/Searchinginsiderecipesusercookbook,enhancing/EnhancingyourusercookbookSSHkeys,distributing/DistributingSSHkeys

usersadding/Addingusersmanaging/Managingusers

VVagrant

about/VagrantandChef-soloinstalling/InstallingVagrantURL/InstallingVagrantused,forprovisioningnewhost/ProvisioninganewhostwithVagrantimage,booting/BootingyourVagrantimagecombining,withChef-solo/CombiningVagrantwithChef-solo,UnderstandingthelimitationsofChef-solo,TellingChef-solowhattorun

VagrantCloudURL/ProvisioninganewhostwithVagrant

Vagrantimagebooting/BootingyourVagrantimage

Wwebapplicationservicerole

defining/Awebapplicationservicerolewebserver

configuring/Configuringthewebserverwebserverrole

creating/Creatingthewebserverrolewhy-runmechanism/Implementingtheproviderworkstation/Terminology,GettingtoknowChef

YYAML/DeployingsoftwarewithChef

the-eye.eu...table of contents chef essentials credits about the author about the reviewers support...

Documents

the-eye.eu · 2017. 6. 8. · table of contents penetration...

+ writing about theatre chapter 5. + writing about theatre...

contentshuettermann.net/devops/devops_ch09+toc.pdf ·...

about the author - jones & bartlett...

the-eye.eu filethe-eye.eu

the-eye.eu food...the-eye.eu

committee & reviewers

reviewers - eprints.hamzanwadi.ac.id

list of suggested reviewers or reviewers not to include...

raspberry pi mechatronics projects...

about the innovation fund innovation fund foundation · two...

the-eye.eu standards solutions... · iv foreword...

angularjs test-driven development -...

winsockapi - the-eye.eu file · web viewwinsockapi -...

the-eye.eu science of vodun voodoo... · the-eye.eu

acknowledgment for reviewers 2010/2011 · acknowledgment...

the-eye.eu history...the-eye.eu

configuration management with...

zabbix network monitoring essentials - omid- · pdf...

the reviewers!