learning docker networking · docker networking options are also covered in this chapter. chapter...

www.allitebooks.com

http://www.allitebooks.org

LearningDockerNetworking

www.allitebooks.com


TableofContents


Credits

AbouttheAuthors

AbouttheReviewer

www.PacktPub.com

eBooks,discountoffers,andmore

Whysubscribe?

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Downloadingthecolorimagesofthisbook

Errata

Piracy

Questions

1.DockerNetworkingPrimer

NetworkingandDocker

Linuxbridges

OpenvSwitch

NAT

IPtables

AppArmor/SELinux

Thedocker0bridge

The—netdefaultmode

The—net=nonemode

www.allitebooks.com


The—net=container:$container2mode

The—net=hostmode

PortmappinginDockercontainer

DockerOVS

Unixdomainsocket

LinkingDockercontainers

Links

What’snewinDockernetworking?

Sandbox

Endpoint

Network

TheDockerCNMmodel

Summary

2.DockerNetworkingInternals

ConfiguringtheIPstackforDocker

IPv4support

IPv6support

ConfiguringaDNSserver

Communicationbetweencontainersandexternalnetworks

RestrictingSSHaccessfromonecontainertoanother

ConfiguringtheDockerbridge

Overlaynetworksandunderlaynetworks

Summary

3.BuildingYourFirstDockerNetwork

IntroductiontoPipework

Multiplecontainersoverasinglehost

Weaveyourcontainers

OpenvSwitch

SinglehostOVS

CreatinganOVSbridge

MultiplehostOVS

www.allitebooks.com


Networkingwithoverlaynetworks–Flannel

Summary

4.NetworkinginaDockerCluster

DockerSwarm

DockerSwarmsetup

DockerSwarmnetworking

Kubernetes

DeployingKubernetesonAWS

KubernetesnetworkinganditsdifferencestoDockernetworking

DeployingtheKubernetespod

Mesosphere

Dockercontainers

DeployingawebappusingDocker

DeployingMesosonAWSusingDCOS

Summary

5.SecurityandQoSforDockerContainers

Filesystemrestrictions

Read-onlymountpoints

sysfs

procfs

/dev/pts

/sys/fs/cgroup

Copy-on-write

Linuxcapabilities

SecuringcontainersinAWSECS

UnderstandingDockersecurityI–kernelnamespaces

pidnamespace

netnamespace

Basicnetworknamespacemanagement

Networknamespaceconfiguration

Usernamespace

www.allitebooks.com


Creatinganewusernamespace

UnderstandingDockersecurityII–cgroups

Definingcgroups

Whyarecgroupsrequired?

Creatingacgroupmanually

Attachingprocessestocgroups

Dockerandcgroups

UsingAppArmortosecureDockercontainers

AppArmorandDocker

Dockersecuritybenchmark

AuditDockerdaemonregularly

Createauserforthecontainer

Donotmountsensitivehostsystemdirectoriesoncontainers

Donotuseprivilegedcontainers

Summary

6.NextGenerationNetworkingStackforDocker:libnetwork

Goal

Design

CNMobjects

Sandbox

Endpoint

Network

Networkcontroller

CNMattributes

CNMlifecycle

Driver

Bridgedriver

Overlaynetworkdriver

UsingoverlaynetworkwithVagrant

OverlaynetworkdeploymentVagrantsetup

OverlaynetworkwithDockerMachineandDockerSwarm

www.allitebooks.com


Prerequisites

Key-valuestoreinstallation

CreateaSwarmclusterwithtwonodes

Creatinganoverlaynetwork

Creatingcontainersusinganoverlaynetwork

Containernetworkinterface

CNIplugin

Networkconfiguration

IPallocation

IPaddressmanagementinterface

ProjectCalico’slibnetworkdriver

Summary

Index

www.allitebooks.com


www.allitebooks.com


LearningDockerNetworkingCopyright©2016PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor(s),norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:February2016

Productionreference:1190216

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78528-095-5

www.packtpub.com

http://www.packtpub.com

CreditsAuthors

RajdeepDua

VaibhavKohli

SantoshKumarKonduri

Reviewer

JonLangemak

CommissioningEditor

KunalParikh

AcquisitionEditor

TusharGupta

ContentDevelopmentEditor

MayurPawanikar

TechnicalEditor

DhirajChandanshive

CopyEditors

JoannaMcMahon

MadhusudanUchil

ProjectCoordinator

NidhiJoshi

Proofreader

SafisEditing

Indexer

RekhaNair

Graphics

JasonMonteiro

ProductionCoordinator

AparnaBhagat

CoverWork

AparnaBhagat

AbouttheAuthorsRajdeepDuahasover16yearsofexperienceindistributedsystems.HehasworkedinR&DandDeveloperRelationrolesatMicrosoft,Google,VMware,andSalesforce.com.HehasexposuretomultiplecloudplatformslikeGoogleAppEngine,Heroku,Force.com,vSphere,andGoogleComputeEngine.

RajdeephasbeenworkingonDockerandrelatedcontainertechnologiesformorethantwoyearsnow.HedidhisMBAinITfromIIMLucknowintheyear2000.

VaibhavKohlihasaround3yearsofworkingexperienceintheresearchanddevelopmentdepartmentofVMware,andhehasbeenteachingcomputerengineeringforayearattheesteemedMumbaiUniversity.HehaspublishedmanyresearchpapersandfiledthreepatentsfromVMwareinthecontainerdomain.Hehasalsoconductedworkshopsinvariouscompaniesandmeetupsoncontainertechnology(Docker)andKubernetes.

SantoshKumarKondurihasaround5yearsofITexperience.HeisanexpertOpenStackadministratorwith3yearsofexperience.

AbouttheReviewerJonLangemakhasover10yearsofexperienceindesigning,building,andmaintaininghigh-performancenetworks.He’scurrentlyemployedasanetworkarchitectataMinnesota-basedcompany,wherehefocusesondisruptivetechnologiesandtheimpacttheyhaveonnetworkoperations.Outsideofwork,Jonblogsatwww.dasblinkenlichten.comandenjoyscollaboratingwithothersinthenetworkingcommunityonnewideasandconcepts.

http://www.dasblinkenlichten.com

www.PacktPub.com

eBooks,discountoffers,andmoreDidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

http://www.PacktPub.com

mailto:[email protected]

http://www.PacktPub.com

https://www2.packtpub.com/books/subscription/packtlib

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

PrefaceThisbookhelpsthereadertolearn,create,deploy,andprovideadministrationstepsforDockernetworking.DockerisaLinuxcontainerimplementationthatenablesthecreationoflight-weightportabledevelopmentandproduction-qualityenvironments.Theseenvironmentscanbeupdatedincrementally.Dockerachievesthisbyleveragingcontainmentprinciples,suchascgroupsandLinuxnamespaces,alongwithoverlayfilesystem-basedportableimages.

Dockerprovidesthenetworkingprimitivesthatallowadministratorstospecifyhowdifferentcontainersnetworkwitheachapplication,connecttoeachoftheircomponents,thendistributethemacrossalargenumberofservers,andensurecoordinationbetweenthemirrespectiveofthehostortheVMthattheyarerunningon.ThisbookaggregatesallthelatestDockernetworkingtechnologyandprovidesgreatindepthexplanationwithsetupdetails.

WhatthisbookcoversChapter1,DockerNetworkingPrimer,explainstheessentialcomponentsofDockernetworking,whichhaveevolvedfromcouplingsimpleDockerabstractionsandpowerfulnetworkcomponents,suchasLinuxbridges,OpenvSwitch,andsoon.ThischapteralsoexplainshowDockercontainerscanbecreatedwithvariousmodes.Inthedefaultmode,portmappinghelpsusthroughtheuseofiptablesNATrules,allowingtrafficarrivingatthehosttoreachcontainers.Laterinthischapter,basiclinkingofthecontaineriscoveredandthenextgenerationofDockernetworking,whichislibnetwork,isalsodiscussed.

Chapter2,DockerNetworkingInternals,discussesDocker’sinternalnetworkingarchitecture.WewilllearnaboutIPv4,IPv6,andDNSconfigurationsinDocker.Laterinthischapter,Dockerbridgeandcommunicationbetweencontainersinsinglehostandmultihostiscovered.ThischapteralsoexplainsoverlaytunnelinganddifferentmethodsthatareimplementedonDockernetworking,suchasOVS,Flannel,andWeave.

Chapter3,BuildingYourFirstDockerNetwork,showshowDockercontainerscommunicatefrommultiplehostsusingdifferentnetworkingoptions,suchasWeave,OVS,andFlannel.PipeworkuseslegacyLinuxbridge,Weavecreatesavirtualnetwork,OVSusesGREtunnelingtechnology,andFlannelprovidesaseparatesubnettoeachhosttoconnectcontainersonmultiplehosts.Someoftheimplementations,suchasPipework,arelegacyandwillbecomeobsoleteoveraperiodoftime,whileothersaredesignedtobeusedinthecontextofspecificOSes,suchasFlannelwithCoreOS.BasiccomparisonsofDockernetworkingoptionsarealsocoveredinthischapter.

Chapter4,NetworkinginaDockerCluster,explainsDockernetworkingindepthusingvariousframeworks,suchasnativeDockerSwarm,whereusingthelibnetworkortheout-ofthe-boxoverlaynetwork,Swarm,providesthemultihostnetworkingfeatures.Kubernetes,ontheotherhand,hasadifferentperspectivefromDocker,whereeachpodwillgetauniqueIPaddressandcommunicationbetweenpodscanoccurwiththehelpofservices.UsingOpenvSwitchorIP-forwardingadvancedroutingrules,theKubernetesnetworkingcanbeenhancedtoprovideconnectivitybetweenpodsondifferentsubnetsacrosshostsandexposethepodstotheexternalworld.InthecaseofMesosphere,wecanseethatMarathonisusedasabackendforthenetworkingofthedeployedcontainers.InthecaseofDCOSoftheMesosphere,theentiredeployedstackofmachinesistreatedasonemachinetoprovidearich-networkingexperiencebetweendeployedcontainerservices.

Chapter5,SecurityandQoSforDockerContainers,takesadiveintoDockersecuritybyreferringtokernelandcgroupsnamespaces.WewillalsovisitsomeoftheaspectsoffilesystemsandvariousLinuxcapabilities,whichcontainersleverageinordertoprovidemorefeatures,suchastheprivilegedcontainerbutatthecostofexposingitselfmoreonthethreatside.WewillalsoseehowcontainerscanbedeployedinasecuredenvironmentinAWSECSusingproxycontainerstorestrictthevulnerabletraffic.WewillalsotalkabouthowAppArmorisalsoprovidedwitharichsetofMandatoryAccessControl(MAC)system,whichprovideskernel-enhancementfeaturesinordertoconfine

applicationstoalimitedsetofresources.LeveragingtheirbenefitstoDockercontainershelpsusdeploytheminasecuredenvironment.Inthelastsection,wetakeaquickdiveintoDockersecuritybenchmarksandsomeoftheimportantrecommendationsthatcanbefollowedduringauditingandDockerdeploymentinaproductionenvironment.

Chapter6,NextGenerationNetworkingStackforDocker:libnetwork,willlookintosomeofthedeeperandconceptualaspectsofDockernetworking.Oneoftheseislibnetworking—thefutureoftheDockernetworkmodel,whichisalreadygettingintoshapewiththereleaseofDocker1.9.Whileexplainingthelibnetworkingconcept,wewillalsostudytheCNMmodel,itsvariousobjectsandcomponents,alongwithitsimplementationcodesnippets.Next,wewilllookintodriversofCNM,theprimeonebeingtheoverlaydriver,indetailwithdeploymentaspartofVagrantsetup.WewilllookatstandaloneintegrationsofcontainerswithoverlaynetworkwithDockerSwarmandDockerMachineaswell.Inthenextsection,weexplaintheCNIinterface,itsexecutableplugins,andgiveatutorialtoconfigureDockernetworkingwiththeCNIplugin.Inthelastsection,ProjectCalicoisexplainedindetail,whichprovidesscalablenetworkingsolutionsthatarebasedoutoflibnetworkandprovidesintegrationwithDocker,Kubernetes,Mesos,bare-metal,andVMs,primarily.

WhatyouneedforthisbookMostlyallofthesetupsbasicallyrequireUbuntu14.04(eitherinstalledonaphysicalmachineorasavirtualmachine)andDocker1.9,whichisthelatestversiontodate.SpecificOSandsoftwarerequirements(OpenSourceGitProjects)arementionedbeforeeachsetupifrequired.

WhothisbookisforIfyouareaLinuxadministratorwhowantstolearnnetworkingusingDockertoensuretheefficientadministrationofcoreelementsandapplications,thenthisbookisforyou.BasicknowledgeofLXC/Dockerisassumed.

www.allitebooks.com


ConventionsYouwillalsofindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“YoumaynoticethatweusedtheUnixcommand,rm,toremovetheDrushdirectoryratherthantheDOSdelcommand.”

Ablockofcodeissetasfollows:

#*FineTuning

#

key_buffer=16M

key_buffer_size=32M

max_allowed_packet=16M

thread_stack=512K

thread_cache_size=8

max_connections=300

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

#*FineTuning

#

key_buffer=16M

key_buffer_size=32M

max_allowed_packet=16M

thread_stack=512K

thread_cache_size=8

max_connections=300

Anycommand-lineinputoroutputiswrittenasfollows:

cd/ProgramData/Propeople

rm-rDrush

gitclone--branchmasterhttp://git.drupal.org/project/drush.git

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,inmenusordialogboxesforexample,appearinthetextlikethis:“OntheSelectDestinationLocationscreen,clickonNexttoacceptthedefaultdestination.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.


http://www.packtpub.com/authors

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesforthisbookfromyouraccountathttp://www.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

Youcandownloadthecodefilesbyfollowingthesesteps:

1. Loginorregistertoourwebsiteusingyoure-mailaddressandpassword.2. HoverthemousepointerontheSUPPORTtabatthetop.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchbox.5. Selectthebookforwhichyou’relookingtodownloadthecodefiles.6. Choosefromthedrop-downmenuwhereyoupurchasedthisbookfrom.7. ClickonCodeDownload.

Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:

WinRAR/7-ZipforWindowsZipeg/iZip/UnRarXforMac7-Zip/PeaZipforLinux

http://www.packtpub.com

http://www.packtpub.com/support

DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefromhttp://www.packtpub.com/sites/default/files/downloads/LearningDockerNetworking_ColorImages.pdf

http://www.packtpub.com/sites/default/files/downloads/LearningDockerNetworking_ColorImages.pdf

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

http://www.packtpub.com/submit-errata

https://www.packtpub.com/books/content/support

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.


QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.


Chapter1.DockerNetworkingPrimerDockerisalightweightcontainertechnologythathasgatheredenormousinterestinrecentyears.ItneatlybundlesvariousLinuxkernelfeaturesandservices,suchasnamespaces,cgroups,SELinux,andAppArmorprofiles,overunionfilesystemssuchasAUFSandBTRFSinordertomakemodularimages.Theseimagesprovideahighlyconfigurablevirtualizedenvironmentforapplicationsandfollowawriteonce,runanywhereworkflow.AnapplicationcanbecomposedofasingleprocessrunninginaDckercontaineroritcouldbemadeupofmultipleprocessesrunningintheirowncontainersandbeingreplicatedastheloadincreases.Therefore,thereisaneedforpowerfulnetworkingelementsthatcansupportvariouscomplexusecases.

Inthischapter,youwilllearnabouttheessentialcomponentsofDockernetworkingandhowtobuildandrunsimplecontainerexamples.

Thischaptercoversthefollowingtopics:

NetworkingandDockerThedocker0bridgenetworkingDockerOVSnetworkingUnixdomainnetworksLinkingDockercontainersWhat’snewinDockernetworking

Dockerisgettingalotoftractionintheindustrybecauseofitsperformance-savvyanduniversalreplicabilityarchitecture,whileprovidingthefollowingfourcornerstonesofmodernapplicationdevelopment:

AutonomyDecentralizationParallelismIsolation

Furthermore,wide-scaleadoptionofThoughtworks’smicroservicesarchitecture,orLOSA(LotsofSmallApplications),isfurtherbringingpotentialtoDockertechnology.Asaresult,bigcompaniessuchasGoogle,VMware,andMicrosofthavealreadyportedDockertotheirinfrastructure,andthemomentumiscontinuedbythelaunchofmyriadDockerstart-ups,namelyTutum,Flocker,Giantswarm,andsoon.

SinceDockercontainersreplicatetheirbehavioranywhere,beityourdevelopmentmachine,abaremetalserver,virtualmachine,ordatacenter,applicationdesignerscanfocustheirattentionondevelopment,whileoperationalsemanticsareleftwithDevOps.Thismakesteamworkflowmodular,efficient,andproductive.Dockerisnottobeconfusedwithavirtualmachine(VM),eventhoughtheyarebothvirtualizationtechnologies.WhileDockersharesanOSwithprovidingasufficientlevelofisolationandsecuritytoapplicationsrunningincontainers,itlatercompletelyabstractsawaytheOSandgivesstrongisolationandsecurityguarantees.However,Docker’sresourcefootprintisminusculeincomparisontoaVMandhencepreferredforeconomyandperformance.

However,itstillcannotcompletelyreplaceVMsandisthereforecomplementarytoVMtechnology.ThefollowingdiagramshowsthearchitectureofVMsandDocker:

NetworkingandDockerEachDockercontainerhasitsownnetworkstack,andthisisduetotheLinuxkernelNETnamespace,whereanewNETnamespaceforeachcontainerisinstantiatedandcannotbeseenfromoutsidethecontainerorfromothercontainers.

Dockernetworkingispoweredbythefollowingnetworkcomponentsandservices.

LinuxbridgesTheseareL2/MAClearningswitchesbuiltintothekernelandaretobeusedforforwarding.

OpenvSwitchThisisanadvancedbridgethatisprogrammableandsupportstunneling.

NATNetworkaddresstranslatorsareimmediateentitiesthattranslateIPaddressesandports(SNAT,DNAT,andsoon).

IPtablesThisisapolicyengineinthekernelusedformanagingpacketforwarding,firewall,andNATfeatures.

AppArmor/SELinuxFirewallpoliciesforeachapplicationcanbedefinedwiththese.

VariousnetworkingcomponentscanbeusedtoworkwithDocker,providingnewwaystoaccessanduseDocker-basedservices.Asaresult,weseealotoflibrariesthatfollowadifferentapproachtonetworking.SomeoftheprominentonesareDockerCompose,Weave,Kubernetes,Pipework,libnetwork,andsoon.ThefollowingfiguredepictstherootideasofDockernetworking:

www.allitebooks.com


Thedocker0bridgeThedocker0bridgeistheheartofdefaultnetworking.WhentheDockerserviceisstarted,aLinuxbridgeiscreatedonthehostmachine.Theinterfacesonthecontainerstalktothebridge,andthebridgeproxiestotheexternalworld.MultiplecontainersonthesamehostcantalktoeachotherthroughtheLinuxbridge.

docker0canbeconfiguredviathe--netflagandhas,ingeneral,fourmodes:

--netdefault

--net=none

--net=container:$container2

--net=host

The—netdefaultmodeInthismode,thedefaultbridgeisusedasthebridgeforcontainerstoconnecttoeachother.

The—net=nonemodeWiththismode,thecontainercreatedistrulyisolatedandcannotconnecttothenetwork.

The—net=container:$container2modeWiththisflag,thecontainercreatedsharesitsnetworknamespacewiththecontainercalled$container2.

The—net=hostmodeWiththismode,thecontainercreatedsharesitsnetworknamespacewiththehost.

PortmappinginDockercontainerInthissection,welookathowcontainerportsaremappedtohostports.ThismappingcaneitherbedoneimplicitlybyDockerEngineorcanbespecified.

IfwecreatetwocontainerscalledContainer1andContainer2,bothofthemareassignedanIPaddressfromaprivateIPaddressspaceandalsoconnectedtothedocker0bridge,asshowninthefollowingfigure:

Boththeprecedingcontainerswillbeabletopingeachotheraswellasreachtheexternalworld.

Forexternalaccess,theirportwillbemappedtoahostport.

Asmentionedintheprevioussection,containersusenetworknamespaces.Whenthefirstcontaineriscreated,anewnetworknamespaceiscreatedforthecontainer.AvEthernetlinkiscreatedbetweenthecontainerandtheLinuxbridge.Trafficsentfrometh0ofthecontainerreachesthebridgethroughthevEthernetinterfaceandgetsswitchedthereafter.ThefollowingcodecanbeusedtoshowalistofLinuxbridges:

#showlinuxbridges

$sudobrctlshow

Theoutputwillbesimilartotheoneshownasfollows,withabridgenameandthevethinterfacesonthecontainersitismappedto:

bridgenamebridgeidSTPenabledinterfaces

docker08000.56847afe9799noveth44cb727

veth98c3700

Howdoesthecontainerconnecttotheexternalworld?Theiptablesnattableonthehostisusedtomasqueradeallexternalconnections,asshownhere:

$sudoiptables-tnat-L–n

...

ChainPOSTROUTING(policyACCEPT)targetprotopt

sourcedestinationMASQUERADEall—172.17.0.0/16

!172.17.0.0/16

...

Howtoreachcontainersfromtheoutsideworld?Theportmappingisagaindoneusingtheiptablesnatoptiononthehostmachine.

DockerOVSOpenvSwitchisapowerfulnetworkabstraction.ThefollowingfigureshowshowOVSinteractswiththeVMs,Hypervisor,andthePhysicalSwitch.EveryVMhasavNICassociatedwithit.EveryvNICisconnectedthroughaVIF(alsocalledavirtualinterface)withtheVirtualSwitch:

OVSusestunnellingmechanismssuchasGRE,VXLAN,orSTTtocreatevirtualoverlaysinsteadofusingphysicalnetworkingtopologiesandEthernetcomponents.ThefollowingfigureshowshowOVScanbeconfiguredforthecontainerstocommunicatebetweenmultiplehostsusingGREtunnels:

UnixdomainsocketWithinasinglehost,UNIXIPCmechanisms,especiallyUNIXdomainsocketsorpipes,canalsobeusedtocommunicatebetweencontainers:

$dockerrun--namec1–v/var/run/foo:/var/run/foo–d–I–tbase

/bin/bash

$dockerrun--namec2–v/var/run/foo:/var/run/foo–d–I–tbase

/bin/bash

Appsonc1andc2cancommunicateoverthefollowingUnixsocketaddress:

structsockaddr_unaddress;

address.sun_family=AF_UNIX;

snprintf(address.sun_path,UNIX_PATH_MAX,"/var/run/foo/bar");

C1:Server.c C2:Client.c

bind(socket_fd,(structsockaddr*)&address,

sizeof(structsockaddr_un));

listen(socket_fd,5);

while((connection_fd=accept(socket_fd,(struct

sockaddr*)&address,&address_length))>-1)

nbytes=read(connection_fd,buffer,256);

connect(socket_fd,(structsockaddr*)

&address,sizeof(structsockaddr_un));

write(socket_fd,buffer,nbytes);

LinkingDockercontainersInthissection,weintroducetheconceptoflinkingtwocontainers.Dockercreatesatunnelbetweenthecontainers,whichdoesn’tneedtoexposeanyportsexternallyonthecontainer.Itusesenvironmentvariablesasoneofthemechanismsforpassinginformationfromtheparentcontainertothechildcontainer.

Inadditiontotheenvironmentvariableenv,Dockeralsoaddsahostentryforthesourcecontainertothe/etc/hostsfile.Thefollowingisanexampleofthehostfile:

$dockerrun-t-i--namec2--rm--linkc1:c1aliastraining/webapp

/bin/bash

root@<container_id>:/opt/webapp#cat/etc/hosts

172.17.0.1aed84ee21bde

...

172.17.0.2c1alaias6e5cdeb2d300c1

Therearetwoentries:

Thefirstisanentryforthecontainerc2thatusestheDockercontainerIDasahostnameThesecondentry,172.17.0.2c1alaias6e5cdeb2d300c1,usesthelinkaliastoreferencetheIPaddressofthec1container

ThefollowingfigureshowstwocontainersContainer1andContainer2connectedusingvethpairstothedocker0bridgewith--icc=true.Thismeansthesetwocontainerscanaccesseachotherthroughthebridge:

LinksLinksprovideservicediscoveryforDocker.Theyallowcontainerstodiscoverandsecurelycommunicatewitheachotherbyusingtheflag-linkname:alias.Inter-containercommunicationcanbedisabledwiththedaemonflag-icc=false.Withthisflagsettofalse,Container1cannotaccessContainer2unlessexplicitlyallowedviaalink.Thisisahugeadvantageforsecuringyourcontainers.Whentwocontainersarelinkedtogether,Dockercreatesaparent-childrelationshipbetweenthem,asshowninthefollowingfigure:

Fromtheoutside,itlookslikethis:

#startthedatabase

$sudodockerrun-dp3306:3306--nametodomvcdb\

-v/data/mysql:/var/lib/mysqlcpswan/todomvc.mysql

#starttheappserver

$sudodockerrun-dp4567:4567--nametodomvcapp\

--linktodomvcdb:dbcpswan/todomvc.sinatra

Ontheinside,itlookslikethis:

$dburl=''mysql://root:pa55Word@''+\ENV[''DB_PORT_3306_TCP_ADDR'']+

''/todomvc''

$DataMapper.setup(:default,dburl)

What’snewinDockernetworking?Dockernetworkingisataverynascentstage,andtherearemanyinterestingcontributionsfromthedevelopercommunity,suchasPipework,Weave,Clocker,andKubernetes.EachofthemreflectsadifferentaspectofDockernetworking.Wewilllearnabouttheminlaterchapters.Docker,Inc.hasalsoestablishedanewprojectwherenetworkingwillbestandardized.Itiscalledlibnetwork.

libnetworkimplementsthecontainernetworkmodel(CNM),whichformalizesthestepsrequiredtoprovidenetworkingforcontainerswhileprovidinganabstractionthatcanbeusedtosupportmultiplenetworkdrivers.TheCNMisbuiltonthreemaincomponents—sandbox,endpoint,andnetwork.

SandboxAsandboxcontainstheconfigurationofacontainer’snetworkstack.Thisincludesmanagementofthecontainer’sinterfaces,routingtable,andDNSsettings.AnimplementationofasandboxcouldbeaLinuxnetworknamespace,aFreeBSDjail,orothersimilarconcept.Asandboxmaycontainmanyendpointsfrommultiplenetworks.

EndpointAnendpointconnectsasandboxtoanetwork.Animplementationofanendpointcouldbeavethpair,anOpenvSwitchinternalport,orsomethingsimilar.Anendpointcanbelongtoonlyonenetworkbutmayonlybelongtoonesandbox.

NetworkAnetworkisagroupofendpointsthatareabletocommunicatewitheachotherdirectly.AnimplementationofanetworkcouldbeaLinuxbridge,aVLAN,andsoon.Networksconsistofmanyendpoints,asshowninthefollowingdiagram:

TheDockerCNMmodelTheCNMprovidesthefollowingcontractbetweennetworksandcontainers:

AllcontainersonthesamenetworkcancommunicatefreelywitheachotherMultiplenetworksarethewaytosegmenttrafficbetweencontainersandshouldbesupportedbyalldriversMultipleendpointspercontainerarethewaytojoinacontainertomultiplenetworksAnendpointisaddedtoanetworksandboxtoprovideitwithnetworkconnectivity

WewilldiscussthedetailsofhowCNMisimplementedinChapter6,NextGenerationNetworkingStackforDocker:libnetwork.

SummaryInthischapter,welearnedabouttheessentialcomponentsofDockernetworking,whichhaveevolvedfromcouplingsimpleDockerabstractionsandpowerfulnetworkcomponentssuchasLinuxbridgesandOpenvSwitch.

WelearnedhowDockercontainerscanbecreatedwithvariousmodes.Inthedefaultmode,portmappinghelpsthroughtheuseofiptablesNATrules,allowingtrafficarrivingatthehosttoreachcontainers.Laterinthechapter,wecoveredthebasiclinkingofcontainers.WealsotalkedaboutthenextgenerationofDockernetworking,whichiscalledlibnetwork.

Chapter2.DockerNetworkingInternalsThischapterdiscussesthesemanticsandsyntaxofDockernetworkingindetail,exposingstrengthsandweaknessesofthecurrentDockernetworkparadigm.

Itcoversthefollowingtopics:

ConfiguringtheIPstackforDocker

IPv4supportIssueswithIPv4addressmanagementIPv6support

ConfiguringDNS

DNSbasicsMulticastDNS

ConfiguringtheDockerbridgeOverlaynetworksandunderlaynetworks

Whatarethey?HowdoesDockerusethem?Whataresomeoftheiradvantages?

ConfiguringtheIPstackforDockerDockerusestheIPstacktointeractwiththeoutsideworldusingTCPorUDP.ItsupportstheIPv4andIPv6addressinginfrastructures,whichareexplainedinthefollowingsubsections.

IPv4supportBydefault,DockerprovidesIPv4addressestoeachcontainer,whichareattachedtothedefaultdocker0bridge.TheIPaddressrangecanbespecifiedwhilestartingtheDockerdaemonusingthe--fixed-cidrflag,asshowninthefollowingcode:

$sudodocker–d--fixed-cidr=192.168.1.0/25

WewilldiscussmoreaboutthisintheConfiguringtheDockerbridgesection.

TheDockerdaemoncanbelistedonanIPv4TCPendpointinadditiontoaUnixsocket:

$sudodocker-Htcp://127.0.0.1:2375-Hunix:///var/run/docker.sock-d&

IPv6supportIPv4andIPv6canruntogether;thisiscalledadualstack.ThisdualstacksupportisenabledbyrunningtheDockerdaemonwiththe--ipv6flag.Dockerwillsetupthedocker0bridgewiththeIPv6link-localaddressfe80::1.Allpacketssharedbetweencontainersflowthroughthisbridge.

ToassigngloballyroutableIPv6addressestoyourcontainers,youhavetospecifyanIPv6subnettopicktheaddressesfrom.

ThefollowingcommandssettheIPv6subnetviathe--fixed-cidr-v6parameterwhilestartingDockerandalsoaddanewroutetotheroutingtable:

#docker–d--ipv6--fixed-cidr-v6="1553:ba3:2::/64"

#dockerrun-t-i--namec0ubuntu:latest/bin/bash

ThefollowingfigureshowsaDockerbridgeconfiguredwithanIPv6addressrange:

IfyouchecktheIPaddressrangeusingifconfiginsideacontainer,youwillnoticethattheappropriatesubnethasbeenassignedtotheeth0interface,asshowninthefollowingcode:

#ifconfig

eth0Linkencap:EthernetHWaddr02:42:ac:11:00:01

inetaddr:172.17.0.1Bcast:0.0.0.0Mask:255.255.0.0

inet6addr:fe80::42:acff:fe11:1/64Scope:Link

inet6addr:1553:ba3:2::242:ac11:1/64Scope:Global

UPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1

RXpackets:7errors:0dropped:0overruns:0frame:0

TXpackets:10errors:0dropped:0overruns:0carrier:0

collisions:0txqueuelen:0

RXbytes:738(738.0B)TXbytes:836(836.0B)

loLinkencap:LocalLoopback

inetaddr:127.0.0.1Mask:255.0.0.0

inet6addr:::1/128Scope:Host

UPLOOPBACKRUNNINGMTU:65536Metric:1





Allthetraffictothe1553:ba3:2::/64subnetwillberoutedviathedocker0interface.

Theprecedingcontainerisassignedusingfe80::42:acff:fe11:1/64asthelink-localaddressand1553:ba3:2::242:ac11:1/64astheglobalroutableIPv6address.

NoteLink-localandloopbackaddresseshavelink-localscope,whichmeanstheyaretobeusedinadirectlyattachednetwork(link).Allotheraddresseshaveglobal(oruniversal)scope,whichmeanstheyaregloballyroutableandcanbeusedtoconnecttoaddresseswithglobalscopeanywhere.

ConfiguringaDNSserverDockerprovideshostnameandDNSconfigurationsforeachcontainerwithoutushavingtobuildacustomimage.Itoverlaysthe/etcfolderinsidethecontainerwithvirtualfiles,inwhichitcanwritenewinformation.

Thiscanbeseenbyrunningthemountcommandinsidethecontainer.Containersreceivethesameresolv.conffileasthatofthehostmachinewhentheyarecreatedinitially.Ifahost’sresolv.conffileismodified,thiswillbereflectedinthecontainer’s/resolv.conffileonlywhenthecontainerisrestarted.

InDocker,youcansetDNSoptionsintwoways:

Usingdockerrun--dns=<ip-address>AddingDOCKER_OPTS="--dnsip-address"totheDockerdaemonfile

Youcanalsospecifythesearchdomainusing--dns-search=<DOMAIN>.

ThefollowingfigureshowsanameserverbeingconfiguredinacontainerusingtheDOCKER_OPTSsettingintheDockerdaemonfile:

ThemainDNSfilesareasfollows:

/etc/hostname

/etc/resolv.conf

/etc/hosts

ThefollowingisthecommandtoaddaDNSserver:

#dockerrun--dns=8.8.8.8--net="bridge"-t-iubuntu:latest/bin/bash

Addhostnamesusingthefollowingcommand:

#dockerrun--dns=8.8.8.8--hostname=docker-vm1-t-iubuntu:latest

/bin/bash

CommunicationbetweencontainersandexternalnetworksPacketscanonlypassbetweencontainersiftheip_forwardparameterissetto1.Usually,youwillsimplyleavetheDockerserveratitsdefaultsetting,--ip-forward=true,andDockerwillsetip_forwardto1foryouwhentheserverstartsup.

TocheckthesettingsortoturnIPforwardingonmanually,usethesecommands:

#cat/proc/sys/net/ipv4/ip_forward

0

#echo1>/proc/sys/net/ipv4/ip_forward

#cat/proc/sys/net/ipv4/ip_forward

1

Byenablingip_forward,userscanmakecommunicationpossiblebetweencontainersandtheexternalworld;itwillalsoberequiredforinter-containercommunicationifyouareinamultiple-bridgesetup.Thefollowingfigureshowshowip_forward=falseforwardsallthepacketsto/fromthecontainerfrom/totheexternalnetwork:

Dockerwillnotdeleteormodifyanypre-existingrulesfromtheDockerfilterchain.Thisallowsuserstocreaterulestorestrictaccesstocontainers.

Dockerusesthedocker0bridgeforpacketflowbetweenallthecontainersonasinglehost.ItaddsaruletoforwardthechainusingIPTablesinorderforthepacketstoflow

betweentwocontainers.Setting--icc=falsewilldropallthepackets.

WhentheDockerdaemonisconfiguredwithboth--icc=falseand--iptables=trueanddockerrunisinvokedwiththe--linkoption,theDockerserverwillinsertapairofIPTablesacceptrulesfornewcontainerstoconnecttotheportsexposedbytheothercontainers,whichwillbetheportsthathavebeenmentionedintheexposedlinesofitsDockerfile.Thefollowingfigureshowshowip_forward=falsedropsallthepacketsto/fromthecontainerfrom/totheexternalnetwork:

Bydefault,Docker’sforwardrulepermitsallexternalIPs.ToallowonlyaspecificIPornetworktoaccessthecontainers,insertanegatedruleatthetopoftheDockerfilterchain.

Forexample,usingthefollowingcommand,youcanrestrictexternalaccesssuchthatonlythesourceIP10.10.10.10canaccessthecontainers:

#iptables–IDOCKER–iext_if!–s10.10.10.10–jDROP

RestrictingSSHaccessfromonecontainertoanotherFollowingthesestepstorestrictSSHaccessfromonecontainertoanother:

1. Createtwocontainers,c1andc2.

Forc1,usethefollowingcommand:

#dockerrun-i-t--namec1ubuntu:latest/bin/bash

Theoutputgeneratedisasfollows:

root@7bc2b6cb1025:/#ifconfig



inet6addr:2001:db8:1::242:ac11:5/64Scope:Global








inetaddr:127.0.0.1Mask:255.0.0.0







Forc2,usethefollowingcommand:


Thefollowingistheoutputgenerated:

root@e58a9bf7120b:/#ifconfig











inetaddr:127.0.0.1Mask:255.0.0.0







WecantestconnectivitybetweenthecontainersusingtheIPaddresswe’vejustdiscovered.Let’sseethisnowusingthepingtool:

root@7bc2b6cb1025:/#ping172.17.0.6

PING172.17.0.6(172.17.0.6)56(84)bytesofdata.

64bytesfrom172.17.0.6:icmp_seq=1ttl=64time=0.139ms


^C

---172.17.0.6pingstatistics---

2packetstransmitted,2received,0%packetloss,time999ms

rttmin/avg/max/mdev=0.110/0.124/0.139/0.018ms

root@7bc2b6cb1025:/#

root@e58a9bf7120b:/#ping172.17.0.5




^C




root@e58a9bf7120b:/#

2. Installopenssh-serveronboththecontainers:

#apt-getinstallopenssh-server

3. Enableiptablesonthehostmachine:

1. Initially,youwillbeabletoSSHfromonecontainertoanother.2. StoptheDockerserviceandaddDOCKER_OPTS="--icc=false--

iptables=true"tothedefaultDockerfileofthehostmachine.Thisoptionwillenabletheiptablesfirewallanddropallportsbetweenthecontainers.

Bydefault,iptablesisnotenabledonthehost.Usethefollowingcommandtoenableit:

root@ubuntu:~#iptables-L-n

ChainINPUT(policyACCEPT)

targetprotoptsourcedestination

ChainFORWARD(policyACCEPT)


DOCKERall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0ctstate

RELATED,ESTABLISHED

ACCEPTall—0.0.0.0/00.0.0.0/0

DOCKERall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0ctstate

RELATED,ESTABLISHED

ACCEPTall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0

#servicedockerstop

#vi/etc/default/docker

3. DockerUpstartandSysVinitconfigurationfile.CustomizethelocationoftheDockerbinary(especiallyfordevelopmenttesting):

#DOCKER="/usr/local/bin/docker"

4. UseDOCKER_OPTStomodifythedaemon’sstartupoptions:

#DOCKER_OPTS="--dns8.8.8.8--dns8.8.4.4"

#DOCKER_OPTS="--icc=false--iptables=true"

5. RestarttheDockerservice:

#servicedockerstart

6. Inspectiptables:






DOCKERall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0ctstateRELATED,

ESTABLISHED

ACCEPTall—0.0.0.0/00.0.0.0/0

DOCKERall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0ctstateRELATED,

ESTABLISHED

ACCEPTall—0.0.0.0/00.0.0.0/0

ACCEPTall—0.0.0.0/00.0.0.0/0

DROPall—0.0.0.0/00.0.0.0/0

TheDROPrulehasbeenaddedtoiptablesonthehostmachine,whichdropsaconnectionbetweencontainers.NowyouwillbeunabletoSSHbetweenthecontainers.

4. Wecancommunicatewithorconnectcontainersusingthe--linkparameter,withthehelpoffollowingsteps:

1. Createthefirstcontainer,whichwillactastheserver,sshserver:

root@ubuntu:~#dockerrun-i-t-p2222:22--namesshserverubuntu

bash

root@9770be5acbab:/#

2. Executetheiptablescommand,andyouwillfindaDockerchainruleadded:

#root@ubuntu:~#iptables-L-n





ChainOUTPUT(policyACCEPT)


ChainDOCKER(0references)


ACCEPTtcp—0.0.0.0/0172.17.0.3tcpdpt:22

3. Createthesecondcontainer,whichactslikeaclient,sshclient:

root@ubuntu:~#dockerrun-i-t--namesshclient--link

sshserver:sshserverubuntubash

root@979d46c5c6a5:/#

4. WecanseethattherearemorerulesaddedtotheDockerchainrule:








ChainDOCKER(0references)


ACCEPTtcp—0.0.0.0/0172.17.0.3tcp

dpt:22

ACCEPTtcp—172.17.0.4172.17.0.3tcp

dpt:22

ACCEPTtcp—172.17.0.3172.17.0.4tcp

spt:22

root@ubuntu:~#

Thefollowingimageexplainscommunicationbetweenthecontainersusingthe--linkflag:

5. Youcaninspectyourlinkedcontainerwiththedockerinspectcommand:

root@ubuntu:~#dockerinspect-f"{{.HostConfig.Links}}"

sshclient

[/sshserver:/sshclient/sshserver]

NowyoucansuccessfullysshintosshserverwithitsIP.

#[email protected]–p22

Usingthe--linkparameter,Dockercreatesasecurechannelbetweenthecontainersthatdoesn’tneedtoexposeanyportsexternallyonthecontainers.

ConfiguringtheDockerbridgeTheDockerservercreatesabridgecalleddocker0bydefaultinsidetheLinuxkernel,anditcanpasspacketsbackandforthbetweenotherphysicalorvirtualnetworkinterfacessothattheybehaveasasingleEthernetnetwork.RunthefollowingcommandtofindoutthelistofinterfacesinaVMandtheIPaddressestheyareconnectedto:

root@ubuntu:~#ifconfig

docker0Linkencap:EthernetHWaddr56:84:7a:fe:97:99


inet6addr:fe80::5484:7aff:fefe:9799/64Scope:Link

inet6addr:fe80::1/64Scope:Link





RXbytes:516868(516.8KB)TXbytes:46460483(46.4MB)

eth0Linkencap:EthernetHWaddr00:0c:29:0d:f4:2c

inetaddr:192.168.186.129Bcast:192.168.186.255

Mask:255.255.255.0

inet6addr:fe80::20c:29ff:fe0d:f42c/64Scope:Link





RXbytes:59902195(59.9MB)TXbytes:3916180(3.9MB)


inetaddr:127.0.0.1Mask:255.0.0.0







Onceyouhaveoneormorecontainersupandrunning,youcanconfirmthatDockerhasproperlyconnectedthemtothedocker0bridgebyrunningthebrctlcommandonthehostmachineandlookingattheinterfacescolumnoftheoutput.

Beforeconfiguringthedocker0bridge,installthebridgeutilities:

#apt-getinstallbridge-utils

Hereisahostwithtwodifferentcontainersconnected:

root@ubuntu:~#brctlshow

bridgenamebridgeidSTPenabledinterfaces

docker08000.56847afe9799noveth21b2e16

veth7092a45

Dockerusesthedocker0bridgesettingswheneveracontaineriscreated.ItassignsanewIPaddressfromtherangeavailableonthebridgewheneveranewcontaineriscreated,ascanbeseenhere:

root@ubuntu:~#dockerrun-t-i--namecontainer1ubuntu:latest/bin/bash

root@e54e9312dc04:/#ifconfig











inetaddr:127.0.0.1Mask:255.0.0.0







root@e54e9312dc04:/#iproute

defaultvia172.17.42.1deveth0

172.17.0.0/16deveth0protokernelscopelinksrc172.17.0.7

Bydefault,Dockerprovidesavirtualnetworkcalleddocker0,whichhastheIPaddress172.17.42.1.DockercontainershaveIPaddressesintherangeof172.17.0.0/16.

TochangethedefaultsettingsinDocker,modifythefile/etc/default/docker.

Changingthedefaultbridgefromdocker0tobr0canbedonelikethis:

#sudoservicedockerstop

#sudoiplinksetdevdocker0down

#sudobrctldelbrdocker0

#sudoiptables-tnat-FPOSTROUTING

#echo'DOCKER_OPTS="-b=br0"'>>/etc/default/docker

#sudobrctladdbrbr0

#sudoipaddradd192.168.10.1/24devbr0

#sudoiplinksetdevbr0up

#sudoservicedockerstart

ThefollowingcommanddisplaysthenewbridgenameandtheIPaddressrangeoftheDockerservice:

root@ubuntu:~#ifconfig

br0Linkencap:EthernetHWaddrae:b2:dc:ed:e6:af


inet6addr:fe80::acb2:dcff:feed:e6af/64Scope:Link






eth0Linkencap:EthernetHWaddr00:0c:29:0d:f4:2c

inetaddr:192.168.186.129Bcast:192.168.186.255

Mask:255.255.255.0

inet6addr:fe80::20c:29ff:fe0d:f42c/64Scope:Link





RXbytes:60081009(60.0MB)TXbytes:4176982(4.1MB)


inetaddr:127.0.0.1Mask:255.0.0.0







OverlaynetworksandunderlaynetworksAnoverlayisavirtualnetworkthatisbuiltontopofunderlyingnetworkinfrastructure(theunderlay).Thepurposeistoimplementanetworkservicethatisnotavailableinthephysicalnetwork.

Networkoverlaydramaticallyincreasesthenumberofvirtualsubnetsthatcanbecreatedontopofthephysicalnetwork,whichinturnsupportsmulti-tenancyandvirtualization.

EverycontainerinDockerisassignedanIPaddress,whichisusedforcommunicationwithothercontainers.Ifacontainerhastocommunicatewiththeexternalnetwork,yousetupnetworkinginthehostsystemandexposeormaptheportfromthecontainertothehostmachine.Withthis,applicationsrunninginsidecontainerswillnotbeabletoadvertisetheirexternalIPandports,astheinformationwillnotbeavailabletothem.

ThesolutionistosomehowassignuniqueIPstoeachDockercontaineracrossallhostsandhavesomenetworkingproductthatroutestrafficbetweenhosts.

TherearedifferentprojectstodealwithDockernetworking,asfollows:

FlannelWeaveOpenvSwitch

FlannelprovidesasolutionbygivingeachcontaineranIPthatcanbeusedforcontainer-to-containercommunication.Usingpacketencapsulation,itcreatesavirtualoverlaynetworkoverthehostnetwork.Bydefault,Flannelprovidesa/24subnettohosts,fromwhichtheDockerdaemonallocatesIPstocontainers.ThefollowingfigureshowsthecommunicationbetweencontainersusingFlannel:

Flannelrunsanagent,flanneld,oneachhostandisresponsibleforallocatingasubnetleaseoutofapreconfiguredaddressspace.Flannelusesetcdtostorethenetworkconfiguration,allocatedsubnets,andauxiliarydata(suchasthehost’sIP).

FlannelusestheuniversalTUN/TAPdeviceandcreatesanoverlaynetworkusingUDPtoencapsulateIPpackets.Subnetallocationisdonewiththehelpofetcd,whichmaintainstheoverlaysubnet-to-hostmappings.

WeavecreatesavirtualnetworkthatconnectsDockercontainersdeployedacrosshosts/VMsandenablestheirautomaticdiscovery.ThefollowingfigureshowsaWeavenetwork:

Weavecantraversefirewallsandoperateinpartiallyconnectednetworks.Trafficcanbeoptionallyencrypted,allowinghosts/VMstobeconnectedacrossanuntrustednetwork.

WeaveaugmentsDocker’sexisting(singlehost)networkingcapabilities,suchasthedocker0bridge,sothesecancontinuetobeusedbycontainers.

OpenvSwitchisanopensourceOpenFlow-capablevirtualswitchthatistypicallyusedwithhypervisorstointerconnectvirtualmachineswithinahostandbetweendifferenthostsacrossnetworks.Overlaynetworksneedtocreateavirtualdatapathusingsupportedtunnelingencapsulations,suchasVXLANandGRE.

TheoverlaydatapathisprovisionedbetweentunnelendpointsresidingintheDockerhost,whichgivestheappearanceofallhostswithinagivenprovidersegmentbeingdirectlyconnectedtooneanother.

Asanewcontainercomesonline,theprefixisupdatedintheroutingprotocol,announcingitslocationviaatunnelendpoint.AstheotherDockerhostsreceivetheupdates,theforwardingruleisinstalledintotheOVSforthetunnelendpointthatthehostresideson.Whenthehostisde-provisioned,asimilarprocessoccursandtunnelendpointDockerhostsremovetheforwardingentryforthede-provisionedcontainer.ThefollowingfigureshowsthecommunicationbetweencontainersrunningonmultiplehoststhroughOVS-basedVXLANtunnels:

SummaryInthischapter,wediscussedDocker’sinternalnetworkingarchitecture.WelearnedaboutIPv4,IPv6,andDNSconfigurationinDocker.Laterinthechapter,wecoveredtheDockerbridgeandcommunicationbetweencontainerswithinasinglehostandinmultiplehosts.

WealsodiscussedoverlaytunnelinganddifferentmethodsthatareimplementedinDockernetworking,suchasOVS,Flannel,andWeave.

Inthenextchapter,wewilllearnhands-onDockernetworking,clubbedwithvariousframeworks.

www.allitebooks.com


Chapter3.BuildingYourFirstDockerNetworkThischapterdescribespracticalexamplesofDockernetworking,spanningmultiplecontainersovermultiplehosts.Wewillcoverthefollowingtopics:

IntroductiontoPipeworkMultiplecontainersovermultiplehostsTowardsscalingnetworks–introducingOpenvSwitchNetworkingwithoverlaynetworks–FlannelComparisonofDockernetworkingoptions

IntroductiontoPipeworkPipeworkletsyouconnecttogethercontainersinarbitrarilycomplexscenarios.

Inpracticalterms,itcreatesalegacyLinuxbridge,addsanewinterfacetothecontainer,andthenattachestheinterfacetothatbridge;containersgetanetworksegmentonwhichtocommunicatewitheachother.

MultiplecontainersoverasinglehostPipeworkisashellscriptandinstallingitissimple:

#sudowget-O/usr/local/bin/pipework

https://raw.githubusercontent.com/jpetazzo/pipework/master/pipework&&sudo

chmod+x/usr/local/bin/pipework

ThefollowingfigureshowscontainercommunicationusingPipework:

First,createtwocontainers:


root@5afb44195a69:/#ifconfig








RXbytes:1038(1.0KB)TXbytes:738(738.0B)


inetaddr:127.0.0.1Mask:255.0.0.0








root@c94d53a76a9b:/#ifconfig










inetaddr:127.0.0.1Mask:255.0.0.0







Nowlet’susePipeworktoconnectthem:

#sudopipeworkbrpipec1192.168.1.1/24

Thiscommandcreatesabridge,brpipe,onthehostmachine.Itaddsaneth1interfacetothecontainerc1withtheIPaddress192.168.1.1andattachestheinterfacetothebridgeasfollows:

root@5afb44195a69:/#ifconfig









eth1Linkencap:EthernetHWaddrce:72:c5:12:4a:1a


inet6addr:fe80::cc72:c5ff:fe12:4a1a/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







#sudopipeworkbrpipec2192.168.1.2/24

Thiscommandwillnotcreatebridgebrpipeasitalreadyexists.Itwilladdaneth1interfacetothecontainerc2andconnectittothebridgeasfollows:

root@c94d53a76a9b:/#ifconfig









eth1Linkencap:EthernetHWaddr36:86:fb:9e:88:ba


inet6addr:fe80::3486:fbff:fe9e:88ba/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







Nowthecontainersareconnectedandwillbeabletopingeachotherastheyareonthesamesubnet,192.168.1.0/24.PipeworkprovidestheadvantageofaddingstaticIPaddressestothecontainers.

WeaveyourcontainersWeavecreatesavirtualnetworkthatcanconnectDockercontainersacrossmultiplehostsasiftheyareallconnectedtoasingleswitch.TheWeaverouteritselfrunsasaDockercontainerandcanencryptroutedtrafficfortransmissionovertheInternet.ServicesprovidedbyapplicationcontainersontheWeavenetworkcanbemadeaccessibletotheoutsideworld,regardlessofwherethosecontainersarerunning.

UsethefollowingcodetoinstallWeave:

#sudocurl-Lgit.io/weave-o/usr/local/bin/weave

#sudochmoda+x/usr/local/bin/weave

ThefollowingfigureshowsmultihostcommunicationusingWeave:

On$HOST1,werunthefollowing:

#weavelaunch

#eval$(weaveproxy-env)

#dockerrun--namec1-tiubuntu

Next,werepeatsimilarstepson$HOST2:

#weavelaunch$HOST1

#eval$(weaveproxy-env)

#dockerrun--namec2-tiubuntu

Inthecontainerstartedon$HOST1,thefollowingoutputisgenerated:

root@c1:/#ifconfig








RXbytes:3166(3.1KB)TXbytes:2299(2.2KB)

ethweLinkencap:EthernetHWaddraa:99:8a:d5:4d:d4


inet6addr:fe80::a899:8aff:fed5:4dd4/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







YoucanseetheWeavenetworkinterface,ethwe,usingtheifconfigcommand:

root@c2:/#ifconfig









ethweLinkencap:EthernetHWaddr8e:7c:17:0d:0e:03


inet6addr:fe80::8c7c:17ff:fe0d:e03/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







#root@c1:/#ping-c1-qc2

PINGc2.weave.local(10.160.0.1)56(84)bytesofdata.

---c2.weave.localpingstatistics---



Similarly,inthecontainerstartedon$HOST2,thefollowingoutputisgenerated:

#root@c2:/#ping-c1-qc1

PINGc1.weave.local(10.128.0.3)56(84)bytesofdata.

---c1.weave.localpingstatistics---



Sotherewehaveit—twocontainersonseparatehostshappilytalkingtoeachother.

OpenvSwitchDockerusestheLinuxbridgedocker0bydefault.However,therearecaseswhereOpenvSwitch(OVS)mightberequiredinsteadofaLinuxbridge.AsingleLinuxbridgecanonlyhandle1024ports–thislimitsthescalabilityofDockeraswecanonlycreate1024containers,eachwithasinglenetworkinterface.

SinglehostOVSWewillnowinstallOVSonasinglehost,createtwocontainers,andconnectthemtoanOVSbridge.

UsethiscommandtoinstallOVS:

#sudoapt-getinstallopenvswitch-switch

Installtheovs-dockerutilitywiththefollowing:

#cd/usr/bin

#wget

https://raw.githubusercontent.com/openvswitch/ovs/master/utilities/ovs-

docker

#chmoda+rwxovs-docker

Thefollowingdiagramshowsthesingle-hostOVS:

CreatinganOVSbridgeHere,wewillbeaddinganewOVSbridgeandconfiguringitsothatwecangetthecontainersconnectedonadifferentnetwork,asfollows:

#ovs-vsctladd-brovs-br1

#ifconfigovs-br1173.16.1.1netmask255.255.255.0up

AddaportfromtheOVSbridgetotheDockercontainerusingthefollowingsteps:

1. CreatetwoUbuntuDockercontainers:

#dockerrun-I-t--namecontainer1ubuntu/bin/bash

#docekrrun-I-t--namecontainer2ubuntu/bin/bash

2. ConnectthecontainertotheOVSbridge:

#ovs-dockeradd-portovs-br1eth1container1--ipaddress=173.16.1.2/24

#ovs-dockeradd-portovs-br1eth1container2--ipaddress=173.16.1.3/24

3. TesttheconnectionbetweenthetwocontainersconnectedviaanOVSbridgeusingthepingcommand.First,findouttheirIPaddresses:

#dockerexeccontainer1ifconfig










inetaddr:127.0.0.1Mask:255.0.0.0







#dockerexeccontainer2ifconfig










inetaddr:127.0.0.1Mask:255.0.0.0







NowthatweknowtheIPaddressesofcontainer1andcontainer2,wecanpingthem:

#dockerexeccontainer2ping172.16.17.2





#dockerexeccontainer1ping172.16.17.2

MultiplehostOVSLet’sseehowtoconnectDockercontainersonmultiplehostsusingOVS.

Let’sconsideroursetupasshowninthefollowingdiagram,whichcontainstwohosts,Host1andHost2,runningUbuntu14.04:

InstallDockerandOpenvSwitchonboththehosts:

#wget-qO-https://get.docker.com/|sh

#sudoapt-getinstallopenvswitch-switch

Installtheovs-dockerutility:

#cd/usr/bin

#wget

https://raw.githubusercontent.com/openvswitch/ovs/master/utilities/ovs-

docker

#chmoda+rwxovs-docker

Bydefault,Dockerchoosesarandomnetworktorunitscontainersin.Itcreatesabridge,docker0,andassignsanIPaddress(172.17.42.1)toit.So,bothHost1andHost2docker0bridgeIPaddressesarethesame,duetowhichitisdifficultforcontainersinboththehoststocommunicate.Toovercomethis,let’sassignstaticIPaddressestothenetwork,thatis,192.168.10.0/24.

Let’sseehowtochangethedefaultDockersubnet.

ExecutethefollowingcommandsonHost1:

#servicedockerstop

#iplinksetdevdocker0down

#ipaddrdel172.17.42.1/16devdocker0

#ipaddradd192.168.10.1/24devdocker0

#iplinksetdevdocker0up

#ipaddrshowdocker0

#servicedockerstart

Addthebr0OVSbridge:

#ovs-vsctladd-brbr0

Createthetunneltotheotherhostandattachittothe:

#add-portbr0gre0—setinterfacegre0type=gre

options:remote_ip=30.30.30.8

Addthebr0bridgetothedocker0bridge:

#brctladdifdocker0br0

ExecutethefollowingcommandsonHost2:

#servicedockerstop

#iptables-tnat-FPOSTROUTING

#iplinksetdevdocker0down

#ipaddrdel172.17.42.1/16devdocker0

#ipaddradd192.168.10.2/24devdocker0

#iplinksetdevdocker0up

#ipaddrshowdocker0

#servicedockerstart

Addthebr0OVSbridge:

#iplinksetbr0up

#ovs-vsctladd-brbr0

Createthetunneltotheotherhostandattachittothe:

#br0bridgeovs-vsctladd-portbr0gre0—setinterfacegre0type=gre

options:remote_ip=30.30.30.7

Addthebr0bridgetothedocker0bridge:

#brctladdifdocker0br0

Thedocker0bridgeisattachedtoanotherbridge,br0.Thistime,it’sanOVSbridge.Thismeansthatalltrafficbetweenthecontainersisroutedthroughbr0too.

Additionally,weneedtoconnecttogetherthenetworksfromboththehostsinwhichthecontainersarerunning.AGREtunnelisusedforthispurpose.Thistunnelisattachedtothebr0OVSbridgeand,asaresult,todocker0too.

Afterexecutingtheprecedingcommandsonbothhosts,youshouldbeabletopingthedocker0bridgeaddressesfrombothhosts.

OnHost1,thefollowingoutputisgeneratedonusingthepingcommand:

#ping192.168.10.2




^C




OnHost2,thefollowingoutputisgeneratedonusingthepingcommand:

#ping192.168.10.1




^C




Let’sseehowtocreatecontainersonboththehosts.

OnHost1,usethefollowingcode:

#dockerrun-t-i--namecontainer1ubuntu:latest/bin/bash

OnHost2,usethefollowingcode:

#dockerrun-t-i--namecontainer2ubuntu:latest/bin/bash

Nowwecanpingcontainer2fromcontainer1.Inthisway,weconnectDockercontainersonmultiplehostsusingOpenvSwitch.

Networkingwithoverlaynetworks–FlannelFlannelisthevirtualnetworklayerthatprovidesthesubnettoeachhostforusewithDockercontainers.ItispackagedwithCoreOSbutcanbeconfiguredonotherLinuxOSesaswell.FlannelcreatestheoverlaybyactuallyconnectingitselftoDockerbridge,towhichcontainersareattached,asshowninthefollowingfigure.TosetupFlannel,twohostmachinesorVMsarerequired,whichcanbeCoreOSor,morepreferably,LinuxOS,asshowninthisfigure:

TheFlannelcodecanbeclonedfromGitHubandbuiltlocally,ifrequired,onadifferentflavorofLinuxOS,asshownhere.ItcomespreinstalledinCoreOS:

#gitclonehttps://github.com/coreos/flannel.git

Cloninginto'flannel'...

remote:Countingobjects:2141,done.

remote:Compressingobjects:100%(19/19),done.

remote:Total2141(delta6),reused0(delta0),pack-reused2122

Receivingobjects:100%(2141/2141),4.

Checkingconnectivity…done.

#sudodockerrun-v`pwd`:/opt/flannel-i-tgoogle/golang/bin/bash-c

"cd/opt/flannel&&./build"

Buildingflanneld…

CoreOSmachinescanbeeasilyconfiguredusingVagrantandVirtualBox,asperthetutorialmentionedinthefollowinglink:

https://coreos.com/os/docs/latest/booting-on-vagrant.html

Afterthemachinesarecreatedandloggedinto,wewillfindaFlannelbridgeautomaticallycreatedusingtheetcdconfiguration:

#ifconfigflannel0

flannel0:flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>mtu1472

inet10.1.30.0netmask255.255.0.0destination10.1.30.0

unspec00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00

txqueuelen500(UNSPEC)

RXpackets243bytes20692(20.2KiB)

RXerrors0dropped0overruns0frame0

TXpackets304bytes25536(24.9KiB)

TXerrors0dropped0overruns0carrier0collisions0

TheFlannelenvironmentcanbecheckedbyviewingsubnet.env:

#cat/run/flannel/subnet.env

FLANNEL_NETWORK=10.1.0.0/16

FLANNEL_SUBNET=10.1.30.1/24

FLANNEL_MTU=1472

FLANNEL_IPMASQ=true

TheDockerdaemonrequirestoberestartedwiththefollowingcommandsinordertogetthenetworkingre-instantiatedwiththesubnetfromtheFlannelbridge:

#source/run/flannel/subnet.env

#sudorm/var/run/docker.pid

#sudoifconfigdocker0${FLANNEL_SUBNET}

#sudodocker-d--bip=${FLANNEL_SUBNET}--mtu=${FLANNEL_MTU}&INFO[0000]

[graphdriver]usingpriorstoragedriver"overlay"

INFO[0000]OptionDefaultDriver:bridge

INFO[0000]OptionDefaultNetwork:bridge

INFO[0000]ListeningforHTTPonunix(/var/run/docker.sock)

INFO[0000]Firewalldrunning:false

INFO[0000]Loadingcontainers:start.

..............

INFO[0000]Loadingcontainers:done.

INFO[0000]Daemonhascompletedinitialization

INFO[0000]Dockerdaemon

commit=cedd534-dirtyexecdriver=native-0.2graphdriver=overlay

version=1.8.3

TheFlannelenvironmentforthesecondhostcanalsobecheckedbyviewingsubnet.env:

#cat/run/flannel/subnet.env

FLANNEL_NETWORK=10.1.0.0/16

FLANNEL_SUBNET=10.1.31.1/24

FLANNEL_MTU=1472

FLANNEL_IPMASQ=true

Adifferentsubnetisallocatedtothesecondhost.TheDockerservicecanalsoberestartedinthishostbypointingtotheFlannelbridge:

#source/run/flannel/subnet.env

#sudoifconfigdocker0${FLANNEL_SUBNET}

https://coreos.com/os/docs/latest/booting-on-vagrant.html

#sudodocker-d--bip=${FLANNEL_SUBNET}--mtu=${FLANNEL_MTU}&INFO[0000]

[graphdriver]usingpriorstoragedriver"overlay"

INFO[0000]ListeningforHTTPonunix(/var/run/docker.sock)

INFO[0000]OptionDefaultDriver:bridge

INFO[0000]OptionDefaultNetwork:bridge



....



INFO[0000]Dockerdaemon

commit=cedd534-dirtyexecdriver=native-0.2graphdriver=overlay

version=1.8.3

Dockercontainerscanbecreatedintheirrespectivehosts,andtheycanbetestedusingthepingcommandinordertochecktheFlanneloverlaynetworkconnectivity.

ForHost1,usethefollowingcommands:

#dockerrun-itubuntu/bin/bash

INFO[0013]POST/v1.20/containers/create

INFO[0013]POST

/v1.20/containers/1d1582111801c8788695910e57c02fdba593f443c15e2f1db9174ed90

78db809/attach?stderr=1&stdin=1&stdout=1&stream=1

INFO[0013]POST


78db809/start

INFO[0013]POST


78db809/resize?h=44&w=80

root@1d1582111801:/#ifconfig

eth0Linkencap:EthernetHWaddr02:42:0a:01:1e:02


inet6addr:fe80::42:aff:fe01:1e02/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







ForHost2,usethefollowingcommands:

#dockerrun-itubuntu/bin/bash

root@ed070166624a:/#ifconfig

eth0Linkencap:EthernetHWaddr02:42:0a:01:1f:02


inet6addr:fe80::42:aff:fe01:1f02/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







root@ed070166624a:/#ping10.1.30.2






Thus,intheprecedingexample,wecanseethecomplexitythatFlannelreducesbyrunningtheflanneldagentoneachhost,whichisresponsibleforallocatingasubnetleaseoutofpreconfiguredaddressspace.Flannelinternallyusesetcdtostorethenetworkconfigurationandotherdetails,suchashostIPandallocatedsubnets.Theforwardingofpacketsisachievedusingthebackendstrategy.

FlannelalsoaimstoresolvetheproblemofKubernetesdeploymentoncloudprovidersotherthanGCE,whereaFlanneloverlaymeshnetworkcaneasetheissueofassigningauniqueIPaddresstoeachpodbycreatingasubnetforeachserver.

SummaryInthischapter,welearnthowDockercontainerscommunicateacrossmultiplehostsusingdifferentnetworkingoptionssuchasWeave,OVS,andFlannel.PipeworkusesthelegacyLinuxbridge,Weavecreatesavirtualnetwork,OVSusesGREtunnelingtechnology,andFlannelprovidesaseparatesubnettoeachhostinordertoconnectcontainerstomultiplehosts.Someoftheimplementations,suchasPipework,arelegacyandwillbecomeobsoleteoveraperiodoftime,whileothersaredesignedtobeusedinthecontextofspecificOSes,suchasFlannelwithCoreOS.

ThefollowingdiagramshowsabasiccomparisonofDockernetworkingoptions:

Inthenextchapter,wewilldiscusshowDockercontainersarenetworkedwhenusingframeworkssuchasKubernetes,DockerSwarm,andMesosphere.

Chapter4.NetworkinginaDockerClusterInthischapter,youwilllearnhowDockercontainersarenetworkedwhenusingframeworkslikeKubernetes,DockerSwarm,andMesosphere.

Wewillcoverthefollowingtopics:

DockerSwarmKubernetes

NetworkedcontainersinaKubernetesclusterHowKubernetesnetworkingdiffersfromDockernetworkingKubernetesonAWS

Mesosphere

DockerSwarmDockerSwarmisanativeclusteringsystemforDocker.DockerSwarmexposesthestandardDockerAPIsothatanytoolthatcommunicateswiththeDockerdaemoncancommunicatewithDockerSwarmaswell.ThebasicaimistoallowthecreationandusageofapoolofDockerhoststogether.TheclustermanagerofSwarmschedulesthecontainersbasedontheavailabilityresourcesinacluster.Wecanalsospecifytheconstrainedresourcesforacontainerwhiledeployingit.Swarmisdesignedtopackcontainersontoahostbysavingotherhostresourcesforheavierandbiggercontainersratherthanschedulingthemrandomlytoahostinthecluster.

SimilartootherDockerprojects,DockerSwarmusesaPlugandPlayarchitecture.DockerSwarmprovidesbackendservicestomaintainalistofIPaddressesinyourSwarmcluster.Thereareseveralservices,suchasetcd,Consul,andZookeeper;evenastaticfilecanbeused.DockerHubalsoprovidesahosteddiscoveryservice,whichisusedinthenormalconfigurationofDockerSwarm.

DockerSwarmschedulingusesmultiplestrategiesinordertoranknodes.Whenanewcontaineriscreated,Swarmplacesitonthenodeonthebasisofthehighestcomputedrank,usingthefollowingstrategies:

1. Spread:Thisoptimizesandschedulesthecontainersonthenodesbasedonthenumberofcontainersrunningonthenodeatthatpointoftime

2. Binpack:ThenodeisselectedtoschedulethecontaineronthebasisofCPUandRAMutilization

3. Randomstrategy:Thisusesnocomputation;itselectsthenoderandomlytoschedulecontainers

DockerSwarmalsousesfiltersinordertoschedulecontainers,suchas:

Constraints:Theseusekey/valuepairsassociatedwithnodes,suchasenvironment=production

Affinityfilter:Thisisusedtorunacontainerandinstructittolocateandrunnexttoanothercontainerbasedonthelabel,image,oridentifierPortfilter:Inthiscase,thenodeisselectedonthebasisoftheportsavailableonitDependencyfilter:Thisco-schedulesdependentcontainersonthesamenodeHealthfilter:Thispreventstheschedulingofcontainersonunhealthynodes

ThefollowingfigureexplainsvariouscomponentsofaDockerSwarmcluster:

DockerSwarmsetupLet’ssetupourDockerSwarmsetup,whichwillhavetwonodesandonemaster.

WewillbeusingaDockerclientinordertoaccesstheDockerSwarmcluster.ADockerclientcanbesetuponamachineorlaptopandshouldhaveaccesstoallthemachinespresentintheSwarmcluster.

AfterinstallingDockeronallthreemachines,wewillrestarttheDockerservicefromacommandlinesothatitcanbeaccessedfromTCPport2375onthelocalhost(0.0.0.0:2375)orfromaspecifichostIPaddressandcanallowconnectionsusingaUnixsocketonalltheSwarmnodes,asfollows:

$docker-Htcp://0.0.0.0:2375-Hunix:///var/run/docker.sock–d&

DockerSwarmimagesarerequiredtobedeployedasDockercontainersonthemasternode.Inourexample,themasternode’sIPaddressis192.168.59.134.ReplaceitwithyourSwarm’smasternode.FromtheDockerclientmachine,wewillbeinstallingDockerSwarmonthemasternodeusingthefollowingcommand:

$sudodocker-Htcp://192.168.59.134:2375run--rmswarmcreate

Unabletofindimage'swarm'locally

Pullingrepositoryswarm

e12f8c5e4c3b:Downloadcomplete

cf43a42a05d1:Downloadcomplete

42c4e5c90ee9:Downloadcomplete

22cf18566d05:Downloadcomplete

048068586dc5:Downloadcomplete

2ea96b3590d8:Downloadcomplete

12a239a7cb01:Downloadcomplete

26b910067c5f:Downloadcomplete

4fdfeb28bd618291eeb97a2096b3f841

TheSwarmtokengeneratedaftertheexecutionofthecommandshouldbenoted,asitwillbeusedfortheSwarmsetup.Inourcase,itisthis:

"4fdfeb28bd618291eeb97a2096b3f841"

Thefollowingarethestepstosetupatwo-nodeDockerSwarmcluster:

1. FromtheDockerclientnode,thefollowingdockercommandisrequiredtobeexecutedwithNode1’sIPaddress(inourcase,192.168.59.135)andtheSwarmtokengeneratedintheprecedingcodeinordertoaddittotheSwarmcluster:

$docker-Htcp://192.168.59.135:2375run-dswarmjoin--

addr=192.168.59.135:2375token://4fdfeb28bd618291eeb97a2096b3f841

Unabletofindimage'swarm'locally

Pullingrepositoryswarm

e12f8c5e4c3b:Downloadcomplete

cf43a42a05d1:Downloadcomplete

42c4e5c90ee9:Downloadcomplete

22cf18566d05:Downloadcomplete

048068586dc5:Downloadcomplete

2ea96b3590d8:Downloadcomplete

12a239a7cb01:Downloadcomplete

26b910067c5f:Downloadcomplete

e4f268b2cc4d896431dacdafdc1bb56c98fed01f58f8154ba13908c7e6fe675b

2. RepeattheprecedingstepsforNode2byreplacingNode1’sIPaddresswithNode2’s.

3. SwarmmanagerisrequiredtobesetuponthemasternodeusingthefollowingcommandontheDockerclientnode:

$sudodocker-Htcp://192.168.59.134:2375run-d-p5001:2375swarm

managetoken://4fdfeb28bd618291eeb97a2096b3f841

f06ce375758f415614dc5c6f71d5d87cf8edecffc6846cd978fe07fafc3d05d3

TheSwarmclusterissetupandcanbemanagedusingtheSwarmmanagerresidingonthemasternode.Tolistallthenodes,thefollowingcommandcanbeexecutedusingaDockerclient:

$sudodocker-Htcp://192.168.59.134:2375run--rmswarmlist\

token://4fdfeb28bd618291eeb97a2096b3f841

192.168.59.135:2375

192.168.59.136:2375

4. Thefollowingcommandcanbeusedtogetinformationaboutthecluster:

$sudodocker-Htcp://192.168.59.134:5001info

Containers:0

Strategy:spread

Filters:affinity,health,constraint,port,dependency

Nodes:2

agent-1:192.168.59.136:2375

└Containers:0

└ReservedCPUs:0/8

└ReservedMemory:0B/1.023GiB

agent-0:192.168.59.135:2375

└Containers:0

└ReservedCPUs:0/8

└ReservedMemory:0B/1.023GiB

5. Thetestubuntucontainercanbelaunchedontotheclusterbyspecifyingthenameasswarm-ubuntuandusingthefollowingcommand:

$sudodocker-Htcp://192.168.59.134:5001run-it--nameswarm-ubuntu

ubuntu/bin/sh

6. ThecontainercanbelistedusingtheSwarmmaster’sIPaddress:

$sudodocker-Htcp://192.168.59.134:5001ps

Thiscompletesthesetupofatwo-nodeDockerSwarmcluster.

DockerSwarmnetworkingDockerSwarmnetworkinghasintegrationwithlibnetworkandevenprovidessupportforoverlaynetworks.libnetworkprovidesaGoimplementationtoconnectcontainers;itisarobustcontainernetworkmodelthatprovidesnetworkabstractionforapplicationsandtheprogramminginterfaceofcontainers.DockerSwarmisnowfullycompatiblewiththenewnetworkingmodelinDocker1.9(notethatwewillbeusingDocker1.9inthefollowingsetup).Thekey-valuestoreisrequiredforoverlaynetworks,whichincludesdiscovery,networks,IPaddresses,andmoreinformation.

Inthefollowingexample,wewillbeusingConsultounderstandDockerSwarmnetworkinginabetterway:

1. WewillprovisionaVirtualBoxmachinecalledsample-keystoreusingdocker-machine:

$docker-machinecreate-dvirtualboxsample-keystore

Runningpre-createchecks…

Creatingmachine…

Waitingformachinetoberunning,thismaytakeafewminutes…

Machineisrunning,waitingforSSHtobeavailable…

Detectingoperatingsystemofcreatedinstance…

Provisioningcreatedinstance…

Copyingcertstothelocalmachinedirectory…

Copyingcertstotheremotemachine…

SettingDockerconfigurationontheremotedaemon…

ToseehowtoconnectDockertothismachine,run:docker-machine.exe

envsample-keystore

2. Wewillalsodeploytheprogrium/consulcontaineronthesample-keystoremachineonport8500withthefollowingcommand:

$docker$(docker-machineconfigsample-keystore)run-d\

-p"8500:8500"\

-h"consul"\

progrium/consul-server–bootstrap

Unabletofindimage'progrium/consul:latest'locally

latest:Pullingfromprogrium/consul

3b4d28ce80e4:Pullcomplete

e5ab901dcf2d:Pullcomplete

30ad296c0ea0:Pullcomplete

3dba40dec256:Pullcomplete

f2ef4387b95e:Pullcomplete

53bc8dcc4791:Pullcomplete

75ed0b50ba1d:Pullcomplete

17c3a7ed5521:Pullcomplete

8aca9e0ecf68:Pullcomplete

4d1828359d36:Pullcomplete

46ed7df7f742:Pullcomplete

b5e8ce623ef8:Pullcomplete

049dca6ef253:Pullcomplete

bdb608bc4555:Pullcomplete

8b3d489cfb73:Pullcomplete

c74500bbce24:Pullcomplete

9f3e605442f6:Pullcomplete

d9125e9e799b:Pullcomplete

Digest:

sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274

Status:Downloadednewerimageforprogrium/consul:latest

1a1be5d207454a54137586f1211c02227215644fa0e36151b000cfcde3b0df7c

3. Setthelocalenvironmenttothesample-keystoremachine:

$eval"$(docker-machineenvsample-keystore)"

4. Wecanlisttheconsulcontainerasfollows:

$dockerps

CONTAINERIDIMAGECOMMANDCREATED

STATUSPORTSNAMES

1a1be5d20745progrium/consul/bin/start-server5minutesagoUp5

minutes53/tcp,53/udp,8300-8302/tcp,8400/tcp,8301-8302/udp,

0.0.0.0:8500->8500/tcpcocky_bhaskara

5. CreateaSwarmclusterusingdocker-machine.ThetwomachinescanbecreatedinVirtualBox;onecanactastheSwarmmaster.AswecreateeachSwarmnode,wewillbepassingtheoptionsrequiredforDockerEnginetohaveanoverlaynetworkdriver:

$docker-machinecreate-dvirtualbox--swarm--swarm-image="swarm"--

swarm-master--swarm-discovery="consul://$(docker-machineipsample-

keystore):8500"--engine-opt="cluster-store=consul://$(docker-machine

ipsample-keystore):8500"--engine-opt="cluster-advertise=eth1:2376"

swarm-master


Creatingmachine…








Configuringswarm…

ToseehowtoconnectDockertothismachine,run:docker-machineenv

swarm-master

Theuseoftheparametersusedintheprecedingcommandisasfollows:

--swarm:ThisisusedtoconfigureamachinewithSwarm.--engine-opt:Thisoptionisusedtodefinearbitrarydaemonoptionsrequiredtobesupplied.Inourcase,wewillsupplytheenginedaemonwiththe--cluster-storeoptionduringcreationtime,whichtellstheenginethelocationofthekey-valuestorefortheoverlaynetworkusability.The--cluster-advertiseoptionwillputthemachineonthenetworkatthespecificport.--swarm-discovery:ItisusedtodiscoverservicestousewithSwarm,inourcase,consulwillbethatservice.--swarm-master:ThisisusedtoconfigureamachineastheSwarmmaster.

6. AnotherhostcanalsobecreatedandaddedtoSwarmcluster,likethis:

$docker-machinecreate-dvirtualbox--swarm--swarm-

image="swarm:1.0.0-rc2"--swarm-discovery="consul://$(docker-machineip

sample-keystore):8500"--engine-opt="cluster-store=consul://$(docker-

machineipsample-keystore):8500"--engine-opt="cluster-

advertise=eth1:2376"swarm-node-1


Creatingmachine…








Configuringswarm…


swarm-node-1

7. Themachinescanbelistedasfollows:

$docker-machinels

NAMEACTIVEDRIVERSTATEURLSWARM

sample-keystore-virtualboxRunning

tcp://192.168.99.100:2376

swarm-master-virtualboxRunning

tcp://192.168.99.101:2376swarm-master(master)

swarm-node-1-virtualboxRunning

tcp://192.168.99.102:2376swarm-master

8. Now,wewillsettheDockerenvironmenttoswarm-master:

$eval$(docker-machineenv--swarmswarm-master)

9. Thefollowingcommandcanbeexecutedonthemasterinordertocreatetheoverlaynetworkandhavemultihostnetworking:

$dockernetworkcreate–driveroverlaysample-net

10. Thenetworkbridgecanbecheckedonthemasterusingthefollowingcommand:

$dockernetworkls

NETWORKIDNAMEDRIVER

9f904ee27bf5sample-netoverlay

7fca4eb8c647bridgebridge

b4234109be9bnonenull

cf03ee007fb4hosthost

11. WhenswitchingtoaSwarmnode,wecaneasilylistthenewlycreatedoverlaynetwork,likethis:

$eval$(docker-machineenvswarm-node-1)

$dockernetworkls

NETWORKIDNAMEDRIVER

7fca4eb8c647bridgebridge

b4234109be9bnonenull

cf03ee007fb4hosthost

9f904ee27bf5sample-netoverlay

12. Oncethenetworkiscreated,wecanstartthecontaineronanyofthehosts,anditwillbepartofthenetwork:

$eval$(docker-machineenvswarm-master)

13. Startthesampleubuntucontainerwiththeconstraintenvironmentsettothefirstnode:

$dockerrun-itd--name=os--net=sample-net--

env="constraint:node==swarm-master"ubuntu

14. Wecancheckusingtheifconfigcommandthatthecontainerhastwonetworkinterfaces,anditwillbeaccessiblefromthecontainerdeployedusingSwarmmanageronanyotherhost.

KubernetesKubernetesisacontainerclustermanagementtool.Currently,itsupportsDockerandRocket.ItisanopensourceprojectsupportedbyGoogle,andtheprojectwaslaunchedinJune2014atGoogleI/O.ItsupportsdeploymentonvariouscloudproviderssuchasGCE,Azure,AWS,andvSphereaswellasonbaremetal.TheKubernetesmanagerislean,portable,extensible,andself-healing.

Kuberneteshasvariousimportantcomponents,asexplainedinthefollowinglist:

Node:Thisisaphysicalorvirtual-machinepartofaKubernetescluster,runningtheKubernetesandDockerservices,ontowhichpodscanbescheduled.Master:ThismaintainstheruntimestateoftheKubernetesserverruntime.ItisthepointofentryforalltheclientcallstoconfigureandmanageKubernetescomponents.Kubectl:Thisisthecommand-linetoolusedtointeractwiththeKubernetesclustertoprovidemasteraccesstoKubernetesAPIs.Throughit,theusercandeploy,delete,andlistpods.Pod:ThisisthesmallestschedulingunitinKubernetes.ItisacollectionofDockercontainersthatsharevolumesanddon’thaveportconflicts.ItcanbecreatedbydefiningasimpleJSONfile.Replicationcontroller:Itmanagesthelifecycleofapodandensuresthataspecifiednumberofpodsarerunningatagiventimebycreatingorkillingpodsasrequired.Label:Labelsareusedtoidentifyandorganizepodsandservicesbasedonkey-valuepairs.

ThefollowingdiagramshowstheKubernetesMaster/Minionflow:

DeployingKubernetesonAWSLet’sgetstartedwithKubernetesclusterdeploymentonAWS,whichcanbedonebyusingtheconfigfilethatalreadyexistsintheKubernetescodebase:

1. LogintoAWSConsoleathttp://aws.amazon.com/console/.2. OpentheIAMconsoleathttps://console.aws.amazon.com/iam/home?#home.3. ChoosetheIAMusername,selecttheSecurityCredentialstab,andclickonthe

CreateAccessKeyoption.4. Afterthekeyshavebeencreated,downloadandkeeptheminasecureplace.The

downloaded.csvfilewillcontainanAccessKeyIDandSecretAccessKey,whichwillbeusedtoconfiguretheAWSCLI.

5. InstallandconfiguretheAWSCLI.Inthisexample,wehaveinstalledAWSCLIonLinuxusingthefollowingcommand:

$sudopipinstallawscli

6. InordertoconfiguretheAWSCLI,usethefollowingcommand:

$awsconfigure

AWSAccessKeyID[None]:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

AWSSecretAccessKey[None]:YYYYYYYYYYYYYYYYYYYYYYYYYYYY

Defaultregionname[None]:us-east-1

Defaultoutputformat[None]:text

7. AfterconfiguringtheAWSCLI,wewillcreateaprofileandattacharoletoitwithfullaccesstoS3andEC2:

$awsiamcreate-instance-profile--instance-profile-nameKube

8. TherolecanbecreatedseparatelyusingtheconsoleorAWSCLIwithaJSONfilethatdefinesthepermissionstherolecanhave:

$awsiamcreate-role--role-nameTest-Role--assume-role-policy-

document/root/kubernetes/Test-Role-Trust-Policy.json

Arolecanbeattachedtotheprecedingprofile,whichwillhavecompleteaccesstoEC2andS3,asshowninthefollowingscreenshot:

http://aws.amazon.com/console/

https://console.aws.amazon.com/iam/home?#home

9. Afterthecreationoftherole,itcanbeattachedtoapolicyusingthefollowingcommand:

$awsiamadd-role-to-instance-profile--role-nameTest-Role--

instance-profile-nameKube

10. Bydefault,thescriptusesthedefaultprofile.Wecanchangeitasfollows:

$exportAWS_DEFAULT_PROFILE=Kube

11. TheKubernetesclustercanbeeasilydeployedusingonecommand,asfollows:

$exportKUBERNETES_PROVIDER=aws;wget-q-O-https://get.k8s.io|

bash

Downloadingkubernetesreleasev1.1.1to/home/vkohli/kubernetes.tar.gz

--2015-11-2210:39:18--https://storage.googleapis.com/kubernetes-

release/release/v1.1.1/kubernetes.tar.gz

Resolvingstorage.googleapis.com(storage.googleapis.com)...

216.58.220.48,2404:6800:4007:805::2010

Connectingtostorage.googleapis.com

(storage.googleapis.com)|216.58.220.48|:443…connected.

HTTPrequestsent,awaitingresponse…200OK

Length:191385739(183M)[application/x-tar]

Savingto:'kubernetes.tar.gz'

100%[======================================>]191,385,7391002KB/sin

3m7s

2015-11-2210:42:25(1002KB/s)-'kubernetes.tar.gz'saved

[191385739/191385739]

Unpackingkubernetesreleasev1.1.1

Creatingakubernetesonaws…

...Startingclusterusingprovider:aws

...callingverify-prereqs

...callingkube-up

Startingclusterusingosdistro:vivid

UploadingtoAmazonS3

Creatingkubernetes-staging-e458a611546dc9dc0f2a2ff2322e724a

make_bucket:s3://kubernetes-staging-e458a611546dc9dc0f2a2ff2322e724a/

+++StagingservertarstoS3Storage:kubernetes-staging-

e458a611546dc9dc0f2a2ff2322e724a/devel

upload:../../../tmp/kubernetes.6B8Fmm/s3/kubernetes-salt.tar.gzto

s3://kubernetes-staging-

e458a611546dc9dc0f2a2ff2322e724a/devel/kubernetes-salt.tar.gz

Completed1of19part(s)with1file(s)remaining

12. Theprecedingcommandwillcallkube-up.shand,inturn,utils.shusingtheconfig-default.shscript,whichcontainsthebasicconfigurationofaK8Sclusterwithfournodes,asfollows:

ZONE=${KUBE_AWS_ZONE:-us-west-2a}

MASTER_SIZE=${MASTER_SIZE:-t2.micro}

MINION_SIZE=${MINION_SIZE:-t2.micro}

NUM_MINIONS=${NUM_MINIONS:-4}

AWS_S3_REGION=${AWS_S3_REGION:-us-east-1}

13. Theinstancesaret2.microrunningUbuntuOS.Theprocesstakes5to10minutes,afterwhichtheIPaddressesofthemasterandminionsgetlistedandcanbeusedto

accesstheKubernetescluster.

KubernetesnetworkinganditsdifferencestoDockernetworkingKubernetesstraysfromthedefaultDockersystem’snetworkingmodel.TheobjectiveisforeachpodtohaveanIPatalevelimpartedbythesystem’sadministrationnamespace,whichhasfullcorrespondencewithotherphysicalmachinesandcontainersoverthesystem.AllocatingIPsperpodunitmakesforaclean,retrogressive,andgoodmodelwhereunitscanbedealtwithmuchlikeVMsorphysicalhostsfromthepointofviewofportallotment,systemadministration,naming,administrationdisclosure,burdenadjustment,applicationdesign,andmovementofpodsfromonehosttoanother.Allcontainersinallpodscanconversewithallothercontainersinallotherpodsusingtheiraddresses.Thisalsohelpsmovetraditionalapplicationstoacontainer-orientedapproach.

AseverypodgetsarealIPaddress,theycancommunicatewitheachotherwithoutanyneedfortranslation.BymakingthesameconfigurationofIPaddressesandportsbothinsideaswellasoutsideofthepod,wecancreateaNAT-lessflataddressspace.ThisisdifferentfromthestandardDockermodelsincethere,allcontainershaveaprivateIPaddress,whichwillallowthemtobeabletoaccessthecontainersonthesamehost.ButinthecaseofKubernetes,allthecontainersinsideapodbehaveasiftheyareonthesamehostandcanreacheachother’sportsonthelocalhost.Thisreducestheisolationbetweencontainersandprovidessimplicity,security,andperformance.Portconflictcanbeoneofthedisadvantagesofthis;thus,twodifferentcontainersinsideonepodcannotusethesameport.

InGCE,usingIPforwardingandadvancedroutingrules,eachVMinaKubernetesclustergetsanextra256IPaddressesinordertoroutetrafficacrosspodseasily.

RoutesinGCEallowyoutoimplementmoreadvancednetworkingfunctionsintheVMs,suchassettingupmany-to-oneNAT.ThisisleveragedbyKubernetes.

ThisisinadditiontothemainEthernetbridgewhichtheVMhas;thisbridgeistermedasthecontainerbridgecbr0inordertodifferentiateitfromtheDockerbridge,docker0.InordertotransferpacketsoutoftheGCEenvironmentfromapod,itshouldundergoanSNATtotheVM’sIPaddress,whichGCErecognizesandallows.

OtherimplementationswiththeprimaryaimofprovidinganIP-per-podmodelareOpenvSwitch,Flannel,andWeave.

InthecaseofaGCE-likesetupofanOpenvSwitchbridgeforKubernetes,themodelwheretheDockerbridgegetsreplacedbykbr0toprovideanextra256subnetaddressesisfollowed.Also,anOVSbridge(ovs0)isadded,whichaddsaporttotheKubernetesbridgeinordertoprovideGREtunnelstotransferpacketsacrossdifferentminionsandconnectpodsresidingonthesehosts.TheIP-per-podmodelisalsoelaboratedmoreintheupcomingdiagram,wheretheserviceabstractionconceptofKubernetesisalsoexplained.

AserviceisanothertypeofabstractionthatiswidelyusedandsuggestedforuseinKubernetesclustersasitallowsagroupofpods(applications)tobeaccessedviavirtualIPaddressesandgetsproxiedtoallinternalpodsinaservice.Anapplicationdeployedin

Kubernetescouldbeusingthreereplicasofthesamepod,whichhavedifferentIPaddresses.However,theclientcanstillaccesstheapplicationontheoneIPaddresswhichisexposedoutside,irrespectiveofwhichbackendpodtakestherequest.Aserviceactsasaloadbalancerbetweendifferentreplicapodsandasinglepointofcommunicationforclientsutilizingthisapplication.Kubeproxy,oneoftheservicesofKubernetes,providesloadbalancingandusesrulestoaccesstheserviceIPsandredirectsthemtothecorrectbackendpod.

DeployingtheKubernetespodNow,inthefollowingexample,wewillbedeployingtwonginxreplicationpods(rc-pod)andexposingthemviaaserviceinordertounderstandKubernetesnetworking.DecidingwheretheapplicationcanbeexposedviaavirtualIPaddressandwhichreplicaofthepod(loadbalancer)therequestistobeproxiedtoistakencareofbyServiceProxy.Pleaserefertothefollowingdiagramformoredetails:

ThefollowingarethestepstodeploytheKubernetespod:

1. IntheKubernetesmaster,createanewfolder:

$mkdirnginx_kube_example

$cdnginx_kube_example

2. Intheeditorofyourchoice,createthe.yamlfilethatwillbeusedtodeploythenginxpods:

$vinginx_pod.yaml

Copythefollowingintothefile:

apiVersion:v1

kind:ReplicationController

metadata:

name:nginx

spec:

replicas:2

selector:

app:nginx

template:

metadata:

name:nginx

labels:

app:nginx

spec:

containers:

-name:nginx

image:nginx

ports:

-containerPort:80

3. Createthenginxpodusingkubectl:

$kubectlcreate-fnginx_pod.yaml

4. Intheprecedingpodcreationprocess,wecreatedtworeplicasofthenginxpod,anditsdetailscanbelistedusingthefollowingcommand:

$kubectlgetpods


NAMEREADYREASONRESTARTSAGE

nginx-karne1/1Running014s

nginx-mo5ug1/1Running014s

Tolistreplicationcontrollersonacluster,usethekubectlgetcommand:

$kubectlgetrc


CONTROLLERCONTAINER(S)IMAGE(S)SELECTORREPLICAS

nginxnginxnginxapp=nginx2

5. Thecontaineronthedeployedminioncanbelistedusingthefollowingcommand:

$dockerps


CONTAINERIDIMAGECOMMAND

CREATEDSTATUSPORTSNAMES

1d3f9cedff1dnginx:latest"nginx-g

'daemonof41secondsagoUp40seconds

k8s_nginx.6171169d_nginx-karne_default_5d5bc813-3166-11e5-8256-

ecf4bb2bbd90_886ddf56

0b2b03b05a8dnginx:latest"nginx-g

'daemonof41secondsagoUp40seconds

6. Deploythenginxserviceusingthefollowing.yamlfileinordertoexposethenginxpodonhostport82:

$vinginx_service.yaml

Copythefollowingintothefile:

apiVersion:v1

kind:Service

metadata:

labels:

name:nginxservice

name:nginxservice

spec:

ports:

#Theportthatthisserviceshouldserveon.

-port:82

#Labelkeysandvaluesthatmustmatchinordertoreceivetraffic

forthisservice.

selector:

app:nginx

type:LoadBalancer

7. Createthenginxserviceusingthekubectlcreatecommand:

$kubectlcreate-fnginx_service.yaml

services/nginxservice

8. Thenginxservicecanbelistedusingthefollowingcommand:

$kubectlgetservices


NAMELABELSSELECTOR

IP(S)PORT(S)

kubernetescomponent=apiserver,provider=kubernetes<none>

192.168.3.1443/TCP

nginxservicename=nginxserviceapp=nginx

192.168.3.4382/TCP

9. Now,thenginxserver’stestpagecanbeaccessedonthefollowingURLviatheservice:http://192.168.3.43:82

MesosphereMesosphereisasoftwaresolutionthatprovideswaysofmanagingserverinfrastructuresandbasicallyexpandsuponthecluster-managementcapabilitiesofApacheMesos.MesospherehasalsolaunchedtheDCOS(datacenteroperatingsystem),usedtomanagedatacentersbyspanningallthemachinesandtreatingthemasasinglecomputer,providingahighlyscalableandelasticwayofdeployingappsontopofit.DCOScanbeinstalledonanypubliccloudoryourownprivatedatacenter,rangingfromAWS,GCE,andMicrosoftAzuretoVMware.MarathonistheframeworkforMesosandisdesignedtolaunchandrunapplications;itservesasareplacementfortheinitsystem.Marathonprovidesvariousfeaturessuchashighavailability,applicationhealthcheck,andservicediscovery,whichhelpyourunapplicationsinMesosclusteredenvironments.

Thissessiondescribeshowtobringupasingle-nodeMesoscluster.

DockercontainersMesoscanrunandmanageDockercontainersusingtheMarathonframework.

Inthisexercise,wewilluseCentOS7todeployaMesoscluster:

1. InstallMesosphereandMarathonusingthefollowingcommand:

#sudorpm-Uvh

http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-

1.noarch.rpm

#sudoyum-yinstallmesosmarathon

ApacheMesosusesZookeepertooperate.ZookeeperactsasthemasterelectionserviceintheMesospherearchitectureandstoresstatesfortheMesosnodes.

2. InstallZookeeperandtheZookeeperserverpackagebypointingtotheRPMrepositoryforZookeeper,asfollows:

#sudorpm-Uvhhttp://archive.cloudera.com/cdh4/one-click-

install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm

#sudoyum-yinstallzookeeperzookeeper-server

3. ValidateZookeeperbystoppingandrestartingit:

#sudoservicezookeeper-serverstop

#sudoservicezookeeper-serverstart

Mesosusesasimplearchitecturetogiveyouintelligenttaskdistributionacrossaclusterofmachineswithoutworryingaboutwheretheyarescheduled.

4. ConfigureApacheMesosbystartingthemesos-masterandmesos-slaveprocessesasfollows:

#sudoservicemesos-masterstart

#sudoservicemesos-slavestart

5. Mesoswillberunningonport5050.Asshowninthefollowingscreenshot,youcanaccesstheMesosinterfacewithyourmachine’sIPaddress,here,http://192.168.10.10:5050:

6. TestMesosusingthemesos-executecommand:

#exportMASTER=$(mesos-resolve`cat/etc/mesos/zk`2>/dev/null)

#mesoshelp

#mesos-execute--master=$MASTER--name="cluster-test"--command="sleep

40"

7. Withthemesos-executecommandrunning,enterCtrl+Ztosuspendthecommand.YoucanseehowitappearsinthewebUIandcommandline:

#hitctrl-z

#mesosps--master=$MASTER

TheMesospherestackusesMarathontomanageprocessesandservices.Itservesasareplacementforthetraditionalinitsystem.Itsimplifiestherunningofapplicationsinaclusteredenvironment.ThefollowingfigureshowstheMesosphereMasterslavetopologywithMarathon:

MarathoncanbeusedtostartotherMesosframeworks;asitisdesignedforlong-runningapplications,itwillensurethattheapplicationsithaslaunchedwillcontinuerunningeveniftheslavenodestheyarerunningonfail.

8. StarttheMarathonserviceusingthefollowingcommand:

#sudoservicemarathonstart

YoucanviewtheMarathonGUIathttp://192.168.10.10:8080.

DeployingawebappusingDockerInthisexercise,wewillinstallasimpleOutyetwebapplication:

1. InstallDockerusingthefollowingcommands:

#sudoyuminstall-ygolanggitdevice-mapper-event-libsdocker

#sudochkconfigdockeron

#sudoservicedockerstart

#exportGOPATH=~/go

#gogetgithub.com/golang/example/outyet

#cd$GOPATH/src/github.com/golang/example/outyet

#sudodockerbuild-toutyet.

2. ThefollowingcommandteststheDockerfilebeforeaddingittoMarathon:

#sudodockerrun--publish6060:8080--nametest--rmoutyet

3. Gotohttp://192.168.10.10:6060/onyourbrowserinordertoconfirmitworks.Onceitdoes,youcanhitCTRL+CtoexittheOutyetDocker.

4. CreateaMarathonapplicationusingMarathonDockersupport,asfollows:

#vi/home/user/outyet.json

{

"id":"outyet",

"cpus":0.2,

"mem":20.0,

"instances":1,

"constraints":[["hostname","UNIQUE",""]],

"container":{

"type":"DOCKER",

"docker":{

"image":"outyet",

"network":"BRIDGE",

"portMappings":[{"containerPort":8080,"hostPort":0,

"servicePort":0,"protocol":"tcp"}

]

}

}

}

#echo'docker,mesos'|sudotee/etc/mesos-slave/containerizers

#sudoservicemesos-slaverestart

5. ContainersareconfiguredandmanagedbetterwithMarathonDocker,asfollows:

#curl-XPOSThttp://192.168.10.10:8080/v2/apps-d

/home/user/outyet.json-H"Content-type:application/json"

6. YoucancheckallyourapplicationsontheMarathonGUIathttp://192.168.10.10:8080,asshowninthefollowingscreenshot:

DeployingMesosonAWSusingDCOSInthisfinalsection,wewillbedeployingthelatestlaunchofDCOSbyMesosphereonAWSinordertomanageanddeployDockerservicesinourdatacenter:

1. CreateanAWSkeypairintheregionwheretheclusterisrequiredtobedeployedbygoingtothenavigationpaneandchoosingKeyPairsunderNETWORK&SECURITY:

2. Afterbeingcreated,thekeycanbeviewedasfollowsandthegeneratedkeypair(.pem)fileshouldbestoredinasecurelocationforfutureuse:

3. TheDCOSclustercanbecreatedbyselectingthe1Mastertemplateontheofficial

Mesospheresite:

ItcanalsobedonebyprovidingthelinkfortheAmazonS3templateURLinthestackdeployment:

4. ClickontheNextbutton.FillinthedetailssuchasStacknameandKeyName,generatedinthepreviousstep:

5. ReviewthedetailsbeforeclickingontheCreatebutton:

6. After5to10minutes,theMesosstackwillbedeployedandtheMesosUIcanbeaccessedattheURLshowninthefollowingscreenshot:

7. Now,wewillbeinstallingtheDCOSCLIonaLinuxmachinewithPython(2.7or3.4)andpippreinstalled,usingthefollowingcommands:

$sudopipinstallvirtualenv

$mkdirdcos

$cddcos

$curl-Ohttps://downloads.mesosphere.io/dcos-cli/install.sh

%Total%Received%XferdAverageSpeedTimeTimeTime

Current

DloadUploadTotalSpentLeft

Speed

1003654100365400363100:00:010:00:01--:--:-

-3635

$ls

install.sh

$bashinstall.sh.http://mesos-dco-elasticl-17lqe4oh09r07-

1358461817.us-west-1.elb.amazonaws.com

InstallingDCOSCLIfromPyPI…

Newpythonexecutablein/home/vkohli/dcos/bin/python

Installingsetuptools,pip,wheel…done.

[core.reporting]:setto'True'

[core.dcos_url]:setto'http://mesos-dco-elasticl-17lqe4oh09r07-

1358461817.us-west-1.elb.amazonaws.com'

[core.ssl_verify]:setto'false'

[core.timeout]:setto'5'

[package.cache]:setto'/home/vkohli/.dcos/cache'

[package.sources]:setto

'[u'https://github.com/mesosphere/universe/archive/version-1.x.zip']'

Gotothefollowinglinkinyourbrowser:

https://accounts.mesosphere.com/oauth/authorize?

scope=&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=c

ode&client_id=6a552732-ab9b-410d-9b7d-d8c6523b09a1&access_type=offline

Enterverificationcode:Skippingauthentication.

Enteremailaddress:Skippingemailinput.

Updatingsource

[https://github.com/mesosphere/universe/archive/version-1.x.zip]

ModifyyourbashprofiletoaddDCOStoyourPATH?[yes/no]yes

FinishedinstallingandconfiguringDCOSCLI.

Runthiscommandtosetupyourenvironmentandtogetstarted:

source~/.bashrc&&dcoshelp

TheDCOShelpfilecanbelistedasfollows:

$source~/.bashrc&&dcoshelp

CommandlineutilityfortheMesosphereDatacenterOperatingSystem

(DCOS).TheMesosphereDCOSisadistributedoperatingsystembuilt

aroundApacheMesos.Thisutilityprovidestoolsforeasymanagementof

aDCOSinstallation.

AvailableDCOScommands:

configGetandsetDCOSCLIconfigurationproperties

helpDisplaycommandlineusageinformation

marathonDeployandmanageapplicationsontheDCOS

nodeManageDCOSnodes

packageInstallandmanageDCOSpackages

serviceManageDCOSservices

taskManageDCOStasks

8. Now,wewilldeployaSparkapplicationontopoftheMesosclusterusingtheDCOSpackageafterupdatingit.Getadetailedcommanddescriptionwithdcos<command>--help:

$dcosconfigshowpackage.sources

[

"https://github.com/mesosphere/universe/archive/version-1.x.zip"

]

$dcospackageupdate

Updatingsource

[https://github.com/mesosphere/universe/archive/version-1.x.zip]

$dcospackagesearch

NAMEVERSIONFRAMEWORKSOURCE

DESCRIPTION

arangodb0.2.1True

https://github.com/mesosphere/universe/archive/version-1.x.zipA

distributedfreeandopen-sourcedatabasewithaflexibledatamodel

fordocuments,graphs,andkey-values.Buildhighperformance

applicationsusingaconvenientSQL-likequerylanguageorJavaScript

extensions.

cassandra0.2.0-1True

https://github.com/mesosphere/universe/archive/version-1.x.zipApache

CassandrarunningonApacheMesos.

chronos2.4.0True

https://github.com/mesosphere/universe/archive/version-1.x.zipAfault

tolerantjobschedulerforMesoswhichhandlesdependenciesandISO8601

basedschedules.

hdfs0.1.7True

https://github.com/mesosphere/universe/archive/version-1.x.zipHadoop

DistributedFileSystem(HDFS),HighlyAvailable.

kafka0.9.2.0True

https://github.com/mesosphere/universe/archive/version-1.x.zipApache

KafkarunningontopofApacheMesos.

marathon0.11.1True

https://github.com/mesosphere/universe/archive/version-1.x.zipA

cluster-wideinitandcontrolsystemforservicesincgroupsorDocker

containers.

spark1.5.0-multi-roles-v2True

https://github.com/mesosphere/universe/archive/version-1.x.zipSpark

isafastandgeneralclustercomputingsystemforBigData.

9. TheSparkpackagecanbeinstalledasfollows:

$dcospackageinstallspark

NotethattheApacheSparkDCOSServiceisbetaandtheremaybebugs,

incompletefeatures,incorrectdocumentationorotherdiscrepancies.

Werecommendaminimumoftwonodeswithatleast2CPUand2GBofRAM

availablefortheSparkServiceandrunningaSparkjob.

Note:TheSparkCLImaytakeupto5mintodownloaddependingonyour

connection.

Continueinstalling?[yes/no]yes

InstallingMarathonappforpackage[spark]version[1.5.0-multi-roles-

v2]

InstallingCLIsubcommandforpackage[spark]version[1.5.0-multi-

roles-v2]

10. Afterdeployment,itcanbeseenintheDCOSUIundertheServicestab,asshowninthefollowingscreenshot:

11. InordertodeployadummyDockerapplicationontheprecedingMarathoncluster,wecanusetheJSONfiletodefinethecontainerimage,commandtoexecute,andportstobeexposedafterdeployment:

$nanodefinition.json

{

"container":{

"type":"DOCKER",

"docker":{

"image":"superguenter/demo-app"

}

},

"cmd":"python-mSimpleHTTPServer$PORT",

"id":"demo",

"cpus":0.01,

"mem":256,

"ports":[3000]

}

12. TheappcanbeaddedtoMarathonandlistedasfollows:

$dcosmarathonappadddefinition.json

$dcosmarathonapplist

IDMEMCPUSTASKSHEALTHDEPLOYMENTCONTAINERCMD

/demo256.00.011/1------DOCKERpython-m

SimpleHTTPServer$PORT

/spark1024.01.01/11/1---DOCKERmv

/mnt/mesos/sandbox/log4j.propertiesconf/log4j.properties&&

./bin/spark-classorg.apache.spark.deploy.mesos.MesosClusterDispatcher

--port$PORT0--webui-port$PORT1--master

mesos://zk://master.mesos:2181/mesos--zkmaster.mesos:2181--host

$HOST--namespark

13. ThreeinstancesoftheprecedingDockerappcanbestartedasfollows:

$dcosmarathonappupdate--forcedemoinstances=3

Createddeployment28171707-83c2-43f7-afa1-5b66336e36d7

$dcosmarathondeploymentlist

APPACTIONPROGRESSID

/demoscale0/128171707-83c2-43f7-afa1-5b66336e36d7

14. ThedeployedapplicationcanbeseenintheDCOSUIbyclickingontheTaskstabunderServices:

SummaryInthischapter,welearntaboutDockernetworkingusingvariousframeworks,suchasthenativeDockerSwarm.Usinglibnetworkorout-of-the-boxoverlaynetworks,Swarmprovidesmultihostnetworkingfeatures.

Kubernetes,ontheotherhand,hasadifferentperspectivefromDocker,inwhicheachpodgetsitsuniqueIPaddressandcommunicationbetweenpodscanoccurwiththehelpofservices.UsingOpenvSwitchorIPforwardingandadvancedroutingrules,Kubernetesnetworkingcanbeenhancedtoprovideconnectivitybetweenpodsondifferentsubnetsacrosshostsandtheabilitytoexposethepodstotheexternalworld.InthecaseofMesosphere,wecanseethatMarathonisusedasthebackendforthenetworkingofthedeployedcontainers.InthecaseofDCOSbyMesosphere,theentiredeployedstackofmachinesistreatedasonemachineinordertoprovidearichnetworkingexperiencebetweendeployedcontainerservices.

Inthenextchapter,wewilllearnaboutsecurityandQoSforbasicDockernetworkingbyunderstandingkernelnamespace,cgroups,andvirtualfirewalls.

Chapter5.SecurityandQoSforDockerContainersInthischapter,wewilllearnhowsecurityisimplementedinthecontextofcontainersingeneralandhowQoSpoliciesareimplementedtomakesurethatresourcessuchasCPUandIOaresharedasintended.MostofthediscussionwillfocusontherelevanceofthesetopicsinthecontextofDocker.

Wewillcoverthefollowinginthischapter:

Filesystemrestrictions

Read-onlymountpointsCopyonwrite

LinuxcapabilitiesandDockerSecuringcontainersinAWSECS(EC2containerservice)UnderstandingDockersecurityI–kernelnamespacesUnderstandingDockersecurityII–cgroupsUsingAppArmourtosecureDockercontainersDockersecuritybenchmark

FilesystemrestrictionsInthissection,wearegoingtostudyfilesystemrestrictionswithwhichDockercontainersarestarted.Thefollowingsectionexplainstheread-onlymountpointsandcopy-on-writefilesystems,whichareusedasabaseforDockercontainersandtherepresentationofkernelobjects.

Read-onlymountpointsDockerneedsaccesstofilesystemssuchassysfsandprocforprocessestofunction.Butitdoesn’tnecessarilyneedtomodifythesemountpoints.

Twoprimarymountpointsloadedinread-onlymodeare:

/sys

/proc

sysfsThesysfsfilesystemisloadedintomountpoint/sys.sysfsisamechanismforrepresentingkernelobjects,theirattributes,andtheirrelationshipswitheachother.Itprovidestwocomponents:

AkernelprogramminginterfaceforexportingtheseitemsviasysfsAuserinterfacetoviewandmanipulatetheseitemsthatmapsbacktothekernelobjectsthattheyrepresent

Thefollowingcodeshowsthemountpointsbeingmounted:

{

Source:"sysfs",

Destination:"/sys",

Device:"sysfs",

Flags:defaultMountFlags|syscall.MS_RDONLY,

},

Areferencelinkfortheprecedingcodeisathttps://github.com/docker/docker/blob/ecc3717cb17313186ee711e624b960b096a9334f/daemon/execdriver/native/template/default_template_linux.go

procfsTheprocfilesystem(procfs)isaspecialfilesysteminUnix-likeoperatingsystems,whichpresentsinformationaboutprocessesandothersystemsinformationinahierarchicalfile-likestructure.Itisloadedinto/proc.Itprovidesamoreconvenientandstandardizedmethodfordynamicallyaccessingprocessdataheldinthekernelthantraditionaltracingmethodsordirectaccesstokernelmemory.Itismappedtoamountpointnamed/procatboottime:

{

Source:"proc",

Destination:"/proc",

Device:"proc",

Flags:defaultMountFlags,

},

Read-onlypathswith/proc:

ReadonlyPaths:[]string{

"/proc/asound",

"/proc/bus",

"/proc/fs",

https://github.com/docker/docker/blob/ecc3717cb17313186ee711e624b960b096a9334f/daemon/execdriver/native/template/default_template_linux.go

"/proc/irq",

"/proc/sys",

"/proc/sysrq-trigger",

}

/dev/ptsThisisanothermountpointthatismountedasread-writeforthecontainerduringcreation./dev/ptslivespurelyinmemoryandnothingisstoredondisk,henceitissafetoloaditinread-writemode.

Entriesin/dev/ptsarepseudo-terminals(ptyforshort).Unixkernelshaveagenericnotionofterminals.Aterminalprovidesawayforapplicationstodisplayoutputandtoreceiveinputthroughaterminaldevice.Aprocessmayhaveacontrollingterminal.Foratextmodeapplication,thisishowitinteractswiththeuser:

{

Source:"devpts",

Destination:"/dev/pts",

Device:"devpts",

Flags:syscall.MS_NOSUID|syscall.MS_NOEXEC,

Data:"newinstance,ptmxmode=0666,mode=0620,gid=5",

},

/sys/fs/cgroupThisisthemountpointwherecgroupsareimplementedandisloadedasMS_RDONLYforthecontainer:

{

Source:"cgroup",

Destination:"/sys/fs/cgroup",

Device:"cgroup",

Flags:defaultMountFlags|syscall.MS_RDONLY,

},

Copy-on-writeDockerusesunionfilesystems,whicharecopy-on-writefilesystems.Thismeanscontainerscanusethesamefilesystemimageasthebaseforthecontainer.Whenacontainerwritescontenttotheimage,itgetswrittentoacontainer-specificfilesystem.Itpreventsonecontainerfrombeingabletoaccessthechangesofanothercontainereveniftheyarecreatedfromthesamefilesystemimage.Onecontainercannotchangetheimagecontenttoeffecttheprocessesinanothercontainer.Thefollowingfigureexplainsthisprocess:

LinuxcapabilitiesDockercontainersbefore1.2couldeitherbegivencompletecapabilitiesunderprivilegedmode,ortheycanallfollowawhitelistofallowedcapabilitieswhiledroppingallothers.Iftheflag--privilegedisused,itwillgrantallcapabilitiestothecontainer.Thiswasnotrecommendedforproductionusebecauseit’sreallyunsafe;itallowedDockerallprivilegesasaprocessunderthedirecthost.

WithDocker1.2,twoflagshavebeenintroducedwithdockerrun:

--cap-add

--cap-drop

Thesetwoflagsprovidefine-graincontroltoacontainer,forexample,asfollows:

ChangethestatusoftheDockercontainer’sinterface:

dockerrun--cap-add=NET_ADMINbusyboxsh-c"iplinketh0down"

PreventanychownintheDockercontainer:

dockerrun--cap-drop=CHOWN…

Allowallcapabilitiesexceptmknod:

dockerrun--cap-add=ALL--cap-drop=MKNOD…

Dockerstartscontainerswitharestrictedsetofcapabilitiesbydefault.Capabilitiesconvertabinarymodeofrootandnon-roottoamorefine-grainedaccesscontrol.Asanexample,awebserverwhichservesHTTPrequestneedstobeboundtoport80forHTTPand443forHTTPs.Theseserversneednotberunintherootmode.Theseserverscanbegrantednet_bind_servicecapability.

Containersandserversarealittledifferentinthiscontext.Serversneedtorunafewprocessesintherootmode.Forexample,ssh,cron,andnetworkconfigurationstohandledhcp,andsoon.Containers,ontheotherhand,donotneedthisaccess.

Thefollowingtasksneednothappeninthecontainer:

sshaccessismanagedbyDockerhostcronjobsshouldberunintheusermodeNetworkconfigurationsuchasipconfigandroutingshouldnothappeninsidethecontainer

Wecansafelydeducecontainersmightnotneedrootpriviledges.

Examplesthatcanbedeniedareasfollows:

DonotallowmountoperationsDonotallowaccesstosocketsPreventaccesstofilesystemoperationssuchaschangingfileattributesorownershipofthefilesPreventthecontainerfromloadingnewmodules

Dockerallowsonlythefollowingcapabilities:

Capabilities:[]string{

"CHOWN",

"DAC_OVERRIDE",

"FSETID",

"FOWNER",

"MKNOD",

"NET_RAW",

"SETGID",

"SETUID",

"SETFCAP",

"SETPCAP",

"NET_BIND_SERVICE",

"SYS_CHROOT",

"KILL",

"AUDIT_WRITE",

},

Areferencetotheprecedingcodeisathttps://github.com/docker/docker/blob/master/daemon/execdriver/native/template/default_template_linux.go

AfulllistofavailablecapabilitiescanbefoundintheLinuxman-pages(http://man7.org/linux/man-pages/man7/capabilities.7.html).

OneprimaryriskwithrunningDockercontainersisthatthedefaultsetofcapabilitiesandmountsgiventoacontainermayprovideincompleteisolation,eitherindependentlyorwhenusedincombinationwithkernelvulnerabilities.

Dockersupportstheadditionandremovalofcapabilities,allowingtheuseofanon-defaultprofile.ThismaymakeDockermoresecurethroughcapabilityremovalorlesssecurethroughtheadditionofcapabilities.Thebestpracticeforuserswouldbetoremoveallcapabilitiesexceptthoseexplicitlyrequiredfortheirprocesses.

https://github.com/docker/docker/blob/master/daemon/execdriver/native/template/default_template_linux.go

http://man7.org/linux/man-pages/man7/capabilities.7.html

SecuringcontainersinAWSECSTheAmazonEC2containerservice(ECS)providesahighlyscalable,high-performancecontainermanagementservicethatsupportsDockercontainers.ItallowsyoutoeasilyrunapplicationsonamanagedclusterofAmazonEC2instances.AmazonECSeliminatestheneedforyoutoinstall,operate,andscaleyourownclustermanagementinfrastructure.WithsimpleAPIcalls,youcanlaunchandstopDocker-enabledapplicationsandquerythecompletestateofyourcluster.

Inthefollowingexample,wewillseehowtodeployasecuredwebapplicationusingtwoDockercontainers,onecontainingasimplewebapplication(applicationcontainer),andtheothercontainingareverseproxywiththrottlingenabled(proxycontainer),whichcanbeusedtoprotectthewebapplication.ThesecontainerswillbedeployedontheAmazonEC2instanceusingECS.Ascanbeseeninthefollowingdiagram,allthenetworktrafficwillberoutedthroughtheproxycontainerthatthrottlesrequests.Also,wecanperformactivitiessuchasfiltering,logging,andintrusiondetectionatproxycontainersusingvarioussecuritysoftware.

Thefollowingarethestepstodoso:

1. WewillbuildabasicPHPwebapplicationcontainerfromtheGitHubproject.ThefollowingstepscanbeperformedonaseparateEC2instanceoralocalmachine:

$sudoyuminstall-ygit

$gitclonehttps://github.com/awslabs/ecs-demo-php-simple-app

2. Changedirectoriestotheecs-demo-php-simple-appfolder:

$cdecs-demo-php-simple-app

3. WecanexamineDockerfileasfollowsinordertounderstandthewebapplicationitwilldeploy:

$catDockerfile

4. BuildthecontainerimageusingDockerfileandthenpushitinyourDockerHubaccount.TheDockerHubaccountisrequiredasithelpstodeploythecontainersontheAmazonECSservicebyjustspecifyingthecontainername:

$dockerbuild-tmy-dockerhub-username/amazon-ecs-sample.

Theimagebuiltoverhereisrequiredtohavedockerhub-username(correctwithoutspaces)asthefirstparameter.

Thefollowingfiguredepictsahackernotabletoaccessthewebapplication,astherequestisfilteredviaaproxycontainerandaccessisblocked:

5. UploadtheDockerimagetotheDockerHubaccount:

$dockerlogin

6. Checktoensureyourloginworked:

$dockerinfo

7. PushyourimagetotheDockerHubaccount:

$dockerpushmy-dockerhub-username/amazon-ecs-sample

8. AftercreatingthesamplewebapplicationDockercontainer,wewillnowcreatetheproxycontainer,whichcanalsocontainsomesecurity-relatedsoftware,ifrequired,inordertostrengthensecurity.WewillcreateanewproxyDockercontainerusingacustomizedDockerfileandthenpushtheimagetoyourDockerHubaccount:

$mkdirproxy-container

$cdproxy-container

$nanoDockerfile

FROMubuntu

RUNapt-getupdate&&apt-getinstall-ynginx

COPYnginx.conf/etc/nginx/nginx.conf

RUNecho"daemonoff;">>/etc/nginx/nginx.conf

EXPOSE80

CMDservicenginxstart

InthepreviousDockerfileweareusingabaseUbuntuimageandinstallingnginxandexposingitonport80.

9. Next,wewillcreateacustomizednginx.conf,whichwilloverridethedefaultnginx.confinordertoensurethereverseproxyisconfiguredproperly:

userwww-data;

worker_processes4;

pid/var/run/nginx.pid;

events{

worker_connections768;

#multi_accepton;

}

http{

server{

listen80;

#Proxypasstoservletcontainer

location/{

proxy_passhttp://application-container:80;

}

}

}

10. BuildtheproxyDockerimageandpushthebuiltimagetotheDockerHubaccount:

$dockerbuild-tmy-dockerhub-username/proxy-image.

$dockerpushmy-dockerhub-username/proxy-image

11. TheECScontainerservicecanbedeployedbynavigatingtoAWSManagementConsole(https://aws.amazon.com/console/).

12. ClickTaskDefinitionsintheleftsidebarandthenclickCreateaNewTaskDefinition.

13. Giveyourtaskdefinitionaname,suchasSecurityApp.14. Next,clickonAddContainerandinsertthenameoftheproxywebcontainer

pushedtotheDockerHubaccount,aswellasthenameoftheapplicationwebcontainer.ViewthecontentsoftheJSONusingConfigureviaJSONtabtoseethetaskdefinitionthatyouhavecreated.Itshouldbelikethis:

Proxy-container:

ContainerName:proxy-container

Image:username/proxy-image

Memory:256

PortMappings

Hostport:80

Containerport:80

Protocol:tcp

CPU:256

Links:application-container

Applicationcontainer:

ContainerName:application-container

Image:username/amazon-ecs-sample

https://aws.amazon.com/console/

Memory:256

CPU:256

ClicktheCreatebuttoninordertodeploytheapplication.

15. ClickClustersintheleftsidebar.Ifadefaultclusterdoesnotexist,createone.16. LaunchanECS-optimizedAmazonMachineImage(AMI),ensuringithasapublic

IPaddressandapathtotheInternet.17. Whenyourinstanceisupandrunning,navigatetotheECSsectionoftheAWS

ManagementConsoleandclickClusters,thendefault.Now,weshouldbeabletoseeourinstanceundertheECSInstancestab.

18. NavigatetotheTASKdefinitionsfromtheleftsideoftheAWSManagementConsoletabandclickRunTask.

19. Onthenextpage,ensuretheclusterissettoDefaultandthenumberoftasksis1,thenclickRunTask.

20. Aftertheprocesscompleteswecanseethestateofthetaskfromapendingstatetoagreenrunningstate.

21. ClickingontheECStab,wecanseethecontainerinstancecreatedearlier.Byclickingonit,wewillgetinformationaboutitspublicIPaddress.ByhittingthispublicIPaddressviathebrowserwewillbeabletoseeoursamplePHPapplication.

UnderstandingDockersecurityI–kernelnamespacesAnamespaceprovidesawrapperaroundaglobalsystemresourceofthekernelandmakestheresourceappeartotheprocesswithinthenamespaceasiftheyhaveanisolatedinstance.Globalresourcechangesarevisibletoprocessesinthesamenamespacebutinvisibletoothers.Containersareconsideredanexcellentimplementationofakernelnamespace.

ThefollowingnamespacesareimplementedbyDocker:

pidnamespace:Usedforprocessisolation(PID—ProcessID)netnamespace:Usedformanagingnetworkinterfaces(NET—Networking)ipcnamespace:UsedformanagingaccesstoIPCresources(IPC—InterProcessCommunication)mntnamespace:Usedformanagingmountpoints(MNT—Mount)utsnamespace:Usedforisolatingkernelandversionidentifiers(UTS—UnixTimesharingSystem)

AddingnamespacesupportinlibcontainerrequiredaddingpatchesinthesystemlayerofGoLang(https://codereview.appspot.com/126190043/patch/140001/150001<emphsis>src/syscall/exec_linux.go</emphsis>sothatnewdatastructurescouldbemaintainedforPIDs,userUIDs,andsoon.

https://codereview.appspot.com/126190043/patch/140001/150001

pidnamespacepidnamespacesisolatetheprocessIDnumberspace;processesindifferentpidnamespacescanhavethesamepid.pidnamespacesallowcontainerstoprovidefunctionalitysuchassuspending/resumingthesetofprocessesinthecontainer,andmigratingthecontainertoanewhostwhiletheprocessesinsidethecontainermaintainthesamepids.

pidsinanewnamespacestartwithPID1.ThekernelneedstobeconfiguredfortheflagCONFIG_PID_NSforthenamespacetowork.

pidnamespacescanbenested.Eachpidnamespacehasaparent,exceptfortheinitial(root)pidnamespace.Theparentofapidnamespaceisthepidnamespaceoftheprocessthatcreatedthenamespaceusingcloneorunshare.pidnamespacesformatree,withallnamespacesultimatelytracingtheirancestrytotherootnamespaceasshowninthefollowingfigure:

netnamespacenetnamespaceprovidesisolationofthesystemresourcesassociatedwithnetworking.Eachnetworknamespacehasitsownnetworkdevices,IPaddresses,IProutingtables,/proc/netdirectory,portnumbers,andsoon.

Networknamespacesmakecontainersusefulfromanetworkingperspective:eachcontainercanhaveitsown(virtual)networkdeviceanditsownapplicationsthatbindtotheper-namespaceportnumberspace;suitableroutingrulesinthehostsystemcandirectnetworkpacketstothenetworkdeviceassociatedwithaspecificcontainer.UseofnetworknamespacesrequiresakernelthatisconfiguredwiththeCONFIG_NET_NSoption(https://lwn.net/Articles/531114/).

Aseachcontainerhasitsownnetworknamespace,whichbasicallymeansitsownnetworkinterfaceandroutingtables,netnamespaceisalsodirectlyleveragedbyDockertoisolateIPaddresses,portnumbers,andsoon.

BasicnetworknamespacemanagementNetworknamespacesarecreatedbypassingaflagtotheclone()systemcall,CLONE_NEWNET.Fromthecommandline,though,itisconvenienttousetheIPnetworkingconfigurationtooltosetupandworkwithnetworknamespaces:

#ipnetnsaddnetns1

Thiscommandcreatesanewnetworknamespacecallednetns1.WhentheIPtoolcreatesanetworknamespace,itwillcreateabindmountforitunder/var/run/netns,whichallowsthenamespacetopersist,evenwhennoprocessesarerunningwithinit,andfacilitatesthemanipulationofthenamespaceitself.Sincenetworknamespacestypicallyrequireafairamountofconfigurationbeforetheyarereadyforuse,thisfeaturewillbeappreciatedbysystemsadministrators.

Theipnetnsexeccommandcanbeusedtorunnetworkmanagementcommandswithinthenamespace:

#ipnetnsexecnetns1iplinklist

1:lo:<LOOPBACK>mtu65536qdiscnoopstateDOWNmodeDEFAULT

link/loopback00:00:00:00:00:00brd00:00:00:00:00:00

Thiscommandliststheinterfacesvisibleinsidethenamespace.Anetworknamespacecanberemovedwiththeuseoffollowingcommand:

#ipnetnsdeletenetns1

Thiscommandremovesthebindmountreferringtothegivennetworknamespace.Thenamespaceitself,however,willpersistforaslongasanyprocessesarerunningwithinit.

NetworknamespaceconfigurationNewnetworknamespaceswillhavealoopbackdevicebutnoothernetworkdevices.Asidefromtheloopbackdevice,eachnetworkdevice(physicalorvirtualinterfaces,bridges,andsoon)canonlybepresentinasinglenetworknamespace.Inaddition,

https://lwn.net/Articles/531114/

physicaldevices(thoseconnectedtorealhardware)cannotbeassignedtonamespacesotherthantheroot.Instead,virtualnetworkdevices(forexample,virtualEthernetorvEth)canbecreatedandassignedtoanamespace.Thesevirtualdevicesallowprocessesinsidethenamespacetocommunicateoverthenetwork;itistheconfiguration,routing,andsoonthatdetermineswhotheycancommunicatewith.

Whenfirstcreated,theloloopbackdeviceinthenewnamespaceisdown,soevenaloopbackpingwillfail.

#ipnetnsexecnetns1ping127.0.0.1

connect:Networkisunreachable

Inthepreviouscommand,wecanseethatsincethenetnamespaceforaDockercontainerisstoredinaseparatelocation,andthusasymlinkisrequiredtobecreatedto/var/run/netns,itcanbedoneinthefollowingway:

#pid=`dockerinspect-f'{{.State.Pid}}'$container_id`

#ln-s/proc/$pid/ns/net/var/run/netns/$container_id

Inthisexample,itisdonebybringingthatinterfaceup,whichwillallowthepingingoftheloopbackaddress.

#ipnetnsexecnetns1iplinksetdevloup








Thisstilldoesn’tallowcommunicationbetweennetns1andtherootnamespace.Todothat,virtualEthernetdevicesneedtobecreatedandconfigured.

#iplinkaddveth0typevethpeernameveth1

#iplinksetveth1netnsnetns1

ThefirstcommandsetsupapairofvirtualEthernetdevicesthatareconnected.Packetssenttoveth0willbereceivedbyveth1andviceversa.Thesecondcommandassignsveth1tothenetns1namespace.

#ipnetnsexecnetns1ifconfigveth110.0.0.1/24up

#ifconfigveth010.0.0.2/24up

Then,thesetwocommandssetIPaddressesforthetwodevices.

#ping10.0.0.1


Communicationinbothdirectionsisnowpossibleasthepreviouspingcommandsshow.

Asmentioned,though,namespacesdonotshareroutingtablesorfirewallrules,asrunningrouteandiptables-Linnetns1willattest:

#ipnetnsexecnetns1route

KernelIProutingtable

DestinationGatewayGenmaskFlagsMetricRefUseIface

10.0.0.0*255.255.255.0U000veth1

#ipnetnsexecnetns1iptables-L







UsernamespaceUsernamespacesallowsper-namespacemappingsofuserandgroupIDs.ThismeansthatuserIDsandgroupIDsofaprocessinsideausernamespacecanbedifferentfromitsIDsoutsideofthenamespace.Aprocesscanhaveanon-zerouserIDoutsideanamespacewhile,atthesametime,havingauserIDofzeroinsidethenamespace.Theprocessisunprivilegedforoperationsoutsidetheusernamespacebuthasrootprivilegesinsidethenamespace.

CreatinganewusernamespaceUsernamespacesarecreatedbyspecifyingtheCLONE_NEWUSERflagwhencallingclone()orunshare():

clone()allowsthechildprocesstosharepartsofitsexecutioncontextwiththecallingprocess,suchasthememoryspace,thetableoffiledescriptors,andthetableofsignalhandlers.

unshare()allowsaprocess(orthread)todisassociatepartsofitsexecutioncontextthatarecurrentlybeingsharedwithotherprocesses(orthreads).Partoftheexecutioncontext,suchasthemountnamespace,issharedimplicitlywhenanewprocessiscreatedusingfork()orvfork().

Asmentionedpreviously,DockercontainersareverysimilartoLXCcontainersasasetofnamespacesandcontrolgroupsarecreatedseparatelyforcontainers.Eachcontainergetsitsownnetworkstackandnamespace.Untilandunlesscontainersdonothavetheprivilegedaccess,theyarenotallowedtoaccessotherhostssocketsorinterfaces.Ifthehostnetworkmodeisgiventothecontainer,thenonlyitgetstheabilitytoaccessthehostportsandIPaddress,whichcancauseapotentialthreattootherprogramsrunningonthehost.

Asshowninthefollowingexample,whereweusethehostnetworkmodeinthecontaineranditisabletoaccessallthehostsbridges:

dockerrun-it--net=hostubuntu/bin/bash

$ifconfig

docker0Linkencap:EthernetHWaddr02:42:1d:36:0d:0d


inet6addr:fe80::42:1dff:fe36:d0d/64Scope:Link

UPBROADCASTMULTICASTMTU:1500Metric:1





eno16777736Linkencap:EthernetHWaddr00:0c:29:02:b9:13

inetaddr:192.168.218.129Bcast:192.168.218.255

Mask:255.255.255.0

inet6addr:fe80::20c:29ff:fe02:b913/64Scope:Link





RXbytes:2909561(2.9MB)TXbytes:577079(577.0KB)

$dockerps-q|xargsdockerinspect--format'{{.Id}}:NetworkMode={{

.HostConfig.NetworkMode}}'

52afb14d08b9271bd96045bebd508325a2adff98dbef8c10c63294989441954d:

NetworkMode=host

Whileauditing,itshouldbecheckedthatallthecontainers,bydefault,havenetworkmodesettodefaultandnothost:

$dockerps-q|xargsdockerinspect--format'{{.Id}}:NetworkMode={{

.HostConfig.NetworkMode}}'

1aca7fe47882da0952702c383815fc650f24da2c94029b5ad8af165239b78968:

NetworkMode=default

EachDockercontainerisconnectedtoanEthernetbridgeinordertoprovideinter-connectivitybetweencontainers.Theycanpingeachothertosend/receiveUDPpacketsandestablishTCPconnections,butthatcanberestrictedifnecessary.Namespacealsoprovidesastraightforwardisolationinrestrictingtheaccessoftheprocessesrunningintheothercontaineraswellasthehost.

Wewillbeusingthefollowingnsentercommandlineutilityinordertoenterintonamespaces.Itisanopen-sourceprojectonGitHubavailableathttps://github.com/jpetazzo/nsenter.

Usingit,wewilltrytoenterexistingcontainernamespacesortrytospawnanewsetofnamespaces.ItisdifferentfromtheDockerexeccommandasnsenterdoesn’tenterthecgroups,whichgivespotentialbenefitsfordebuggingandexternalauditsbyescapingtheresourcelimitationsusingnamespace.

WecaninstallnsenterfromPyPI(itrequiresPython3.4)andusethecommandlineutilitytoconnecttoarunningcontainer:

$pipinstallnsenter

Toreplacepidwiththecontainer’spid,usethefollowingcommand:

$sudonsenter--net--target=PID/bin/ipa

1:lo:<LOOPBACK,UP,LOWER_UP>mtu65536qdiscnoqueuestateUNKNOWNgroup

default

link/loopback00:00:00:00:00:00brd00:00:00:00:00:00

inet127.0.0.1/8scopehostlo

valid_lftforeverpreferred_lftforever

inet6::1/128scopehost


14:eth0:<BROADCAST,UP,LOWER_UP>mtu1500qdiscnoqueuestateUPgroup

default

link/ether02:42:ac:11:00:06brdff:ff:ff:ff:ff:ff

inet172.17.0.6/16scopeglobaleth0


inet6fe80::42:acff:fe11:6/64scopelink


Wecanusethedockerinspectcommandtomakeitmoreconvenientasfollows:

https://github.com/jpetazzo/nsenter

1. Firststartanewnginxserver:

$dockerrun-d--name=nginx-tnginx

2. Thengetpidofthecontainer:

PID=$(dockerinspect--format{{.State.Pid}}nginx)

3. Connecttotherunningnginxcontainer:

$nsenter--target$PID--uts--ipc--net–pid

docker-enterisalsooneofthewrappersthatcanbeusedtoenterinsidethecontainerspecifyingtheshellcommands,andifnocommandisspecified,ashellwillbeinvokedinstead.Ifitisrequiredtoinspectormanipulatecontainerswithoutexecutinganothercommandlinetool,wecanusecontextmanagertodothis:

importsubprocess

fromnsenterimportNamespace

withNamespace(mypid,'net'):

#outputnetworkinterfacesasseenfromwithinthemypid'snetNS:

subprocess.check_output(['ip','a'])

UnderstandingDockersecurityII–cgroupsInthissection,welookathowcgroupsformthebackboneofisolationforacontainer.

DefiningcgroupsControlgroupsprovideamechanismforaggregating/partitioningsetsoftasks(processes),andalltheirfuturechildren,intohierarchicalgroups.

Acgroupassociatesasetataskswithparametersfromasubsystem.Asubsystemitselfisaresourcecontrollerusedtodefineboundariesforcgroupsorforprovisioningaresource.

Ahierarchyisasetofcgroupsarrangedinatree,suchthateverytaskinthesystemisinexactlyoneofthecgroupsinthehierarchyandasetofsubsystems.

Whyarecgroupsrequired?TherearemultipleeffortstoprovideprocessaggregationsintheLinuxkernel,mainlyforresource-trackingpurposes.

Sucheffortsincludecpusets,CKRM/ResGroups,UserBeanCounters,andvirtualservernamespaces.Theseallrequirethebasicnotionofagrouping/partitioningofprocesses,withnewlyforkedprocessesendingupinthesamegroup(cgroup)astheirparentprocess.

Thekernelcgrouppatchprovidesessentialkernelmechanismstoefficientlyimplementsuchgroups.Ithasminimalimpactonthesystemfastpathsandprovideshooksforspecificsubsystemssuchascpusetstoprovideadditionalbehaviorasdesired.

CreatingacgroupmanuallyInthefollowingsteps,wewillcreateacpusetcontrolgroup:

#mount-ttmpfscgroup_root/sys/fs/cgroup

tmpfsisafilesystemthatkeepsallfilesinvirtualmemory.Everythingintmpfsistemporaryinthesensethatnofileswillbecreatedonyourharddrive.Ifyouunmountatmpfsinstance,everythingstoredthereinislost:

#mkdir/sys/fs/cgroup/cpuset

#mount-tcgroup-ocpusetcpuset/sys/fs/cgroup/cpuset

#cd/sys/fs/cgroup/cpuset

#mkdirCharlie

#cdCharlie

#ls

cgroup.clone_childrencpuset.cpu_exclusivecpuset.mem_hardwall

cpuset.memory_spread_pagecpuset.sched_load_balancetasks

cgroup.event_controlcpuset.cpuscpuset.memory_migrate

cpuset.memory_spread_slabcpuset.sched_relax_domain_level

cgroup.procscpuset.mem_exclusivecpuset.memory_pressure

cpuset.memsnotify_on_release

AssignCPUandmemorylimitstothiscgroup:

#/bin/echo2-3>cpuset.cpus

#/bin/echo0>cpuset.mems

#/bin/echo$$>tasks

Thefollowingcommandshows/Charlieasthecpusetcgroup:

#cat/proc/self/cgroup

11:name=systemd:/user/1000.user/c2.session

10:hugetlb:/user/1000.user/c2.session

9:perf_event:/user/1000.user/c2.session

8:blkio:/user/1000.user/c2.session

7:freezer:/user/1000.user/c2.session

6:devices:/user/1000.user/c2.session

5:memory:/user/1000.user/c2.session

4:cpuacct:/user/1000.user/c2.session

3:cpu:/user/1000.user/c2.session

2:cpuset:/Charlie

AttachingprocessestocgroupsAddtheprocessIDPID{X}tothetasksfileasshowninthefollowing:

#/bin/echoPID>tasks

NotethatitisPID,notPIDs.

Youcanonlyattachonetaskatatime.Ifyouhaveseveraltaskstoattach,youhavetodoitoneafteranother:

#/bin/echoPID1>tasks

#/bin/echoPID2>tasks

...

#/bin/echoPIDn>tasks

Attachthecurrentshelltaskbyechoing0:

#echo0>tasks

DockerandcgroupscgroupsaremanagedaspartofthelibcontainerprojectunderDocker’sGitHubrepo(https://github.com/opencontainers/runc/tree/master/libcontainer/cgroups).ThereisacgroupmanagerthatmanagestheinteractionwiththecgroupAPIsinthekernel.

Thefollowingcodeshowsthelifecycleeventsmanagedbythemanager:

typeManagerinterface{

//Applycgroupconfigurationtotheprocesswiththespecifiedpid

Apply(pidint)error

//ReturnsthePIDsinsidethecgroupset

GetPids()([]int,error)

//Returnsstatisticsforthecgroupset

GetStats()(*Stats,error)

//Togglesthefreezercgroupaccordingwithspecifiedstate

Freeze(stateconfigs.FreezerState)error

//Destroysthecgroupset

Destroy()error

//Pathsmapscgroupsubsystemtopathatwhichitismounted.

//Cgroupsspecifiesspecificcgroupsettingsforthevarioussubsystems

//Returnscgrouppathstosaveinastatefileandtobeableto

//restoretheobjectlater.

GetPaths()map[string]string

//Setthecgroupasconfigured.

Set(container*configs.Config)error

}

https://github.com/opencontainers/runc/tree/master/libcontainer/cgroups

UsingAppArmortosecureDockercontainersAppArmorisaMandatoryAccessControl(MAC)systemthatisakernelenhancementtoconfineprogramstoalimitedsetofresources.AppArmor’ssecuritymodelistobindaccesscontrolattributestoprogramsratherthantousers.

AppArmorconfinementisprovidedviaprofilesloadedintothekernel,typicallyonboot.AppArmorprofilescanbeinoneoftwomodes:enforcementorcomplain.

Profilesloadedinenforcementmodewillresultinenforcementofthepolicydefinedintheprofile,aswellasreportingpolicyviolationattempts(eitherviasyslogorauditd).

Profilesincomplainmodewillnotenforcepolicybutinsteadreportpolicyviolationattempts.

AppArmordiffersfromsomeotherMACsystemsonLinux:itispath-based,itallowsmixingofenforcementandcomplain-modeprofiles,itusesincludefilestoeasedevelopment,andithasafarlowerbarriertoentrythanotherpopularMACsystems.ThefollowingfigureshowstheAppArmourapplicationprofileslinkedtoapps:

AppArmorisanestablishedtechnologyfirstseeninImmunixandlaterintegratedintoUbuntu,Novell/SUSE,andMandriva.CoreAppArmorfunctionalityisinthemainlineLinuxkernelfrom2.6.36onwards;workisongoingbyAppArmor,Ubuntu,andotherdeveloperstomergeadditionalAppArmorfunctionalityintothemainlinekernel.

YoucanfindmoreinformationaboutAppArmorathttps://wiki.ubuntu.com/AppArmor.

https://wiki.ubuntu.com/AppArmor

AppArmorandDockerApplicationsrunninginsideDockercanleverageAppArmorfordefiningpolicies.Theseprofilescaneitherbecreatedmanuallyorloadedusingatoolcalledbane.

NoteOnUbuntu14.x,makesuresystemdisinstalledforthefollowingcommandstowork.

Thefollowingstepsshowhowtousethistool:

1. DownloadthebaneprojectforGitHub:

$gitclonehttps://github.com/jfrazelle/bane

MakesurethisisdoneinthedirectoryinyourGOPATH.Forexample,weused/home/ubuntu/goandthebanesourcewasdownloadedin/home/Ubuntu/go/src/github.com/jfrazelle/bane.

2. Installtomlparserneededbybanetobecompiled:

$gogetgithub.com/BurntSushi/toml

3. Gotothe/home/Ubuntu/go/src/github.com/jfrazelle/banedirectoryandrunthefollowingcommand:

$goinstall

4. Youwillfindthebanebinaryin/home/Ubuntu/go/bin.5. Usea.tomlfiletocreateaprofile:

Name="nginx-sample"

[Filesystem]

#readonlypathsforthecontainer

ReadOnlyPaths=[

"/bin/**",

"/boot/**",

"/dev/**",

"/etc/**",

…

]

AllowExec=[

"/usr/sbin/nginx"

]

#deniedexecutablefiles

DenyExec=[

"/bin/dash",

"/bin/sh",

"/usr/bin/top"

]

6. Executebanetoloadtheprofile.sample.tomlisafileinthedirectory/home/Ubuntu/go/src/github.com/jfrazelle/bane:

$sudobanesample.toml

#Profileinstalledsuccessfullyyoucannowruntheprofilewith#

`dockerrun--security-opt="apparmor:docker-nginx-sample"`

Thisprofilewillmakeawholelotofpathsreadonlyandallowsonlynginxexecutioninthecontainerwearegoingtocreate.ItdisablesTOP,PING,andsoon.

7. Oncetheprofileisloadedyoucancreateanginxcontainer:

$dockerrun--security-opt="apparmor:docker-nginx-sample"-p80:80--

rm-itnginxbash

Note,ifAppArmorisnotabletofindthefile,copythefileintothe/etc/apparmor.ddirectoryandreloadtheAppArmourprofiles:

$sudoinvoke-rc.dapparmorreload

CreatethenginxcontainerwiththeAppArmorprofile:

ubuntu@ubuntu:~/go/src/github.com$dockerrun--security-

opt="apparmor:docker-nginx-sample"-p80:80--rm-itnginxbash

root@84d617972e04:/#ping8.8.8.8

ping:Lackingprivilegeforrawsocket.

ThefollowingfigureshowshowannginxapprunninginsideacontainerusesAppArmourapplicationprofiles:

DockersecuritybenchmarkThefollowingtutorialshowssomeoftheimportantguidelinesthatshouldbefollowedinordertorunDockercontainersinsecuredandproductionenvironments.ItisreferredfromtheCISDockerSecurityBenchmarkhttps://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.6_Benchmark_v1.0.0.pdf.

AuditDockerdaemonregularlyApartfromauditingyourregularLinuxfilesystemandsystemcalls,auditDockerdaemonaswell.Dockerdaemonrunswithrootprivileges.Itisthusnecessarytoaudititsactivitiesandusage:

$apt-getinstallauditd

Readingpackagelists…Done

Buildingdependencytree

Readingstateinformation…Done

Thefollowingextrapackageswillbeinstalled:

libauparse0

Suggestedpackages:

audispd-plugins

ThefollowingNEWpackageswillbeinstalled:

auditdlibauparse0

0upgraded,2newlyinstalled,0toremoveand50notupgraded.

Processingtriggersforlibc-bin(2.21-0ubuntu4)...

Processingtriggersforureadahead(0.100.0-19)...

Processingtriggersforsystemd(225-1ubuntu9)...

Removetheauditlogfile,ifitexists:

$cd/etc/audit/

$ls

audit.log

$nanoaudit.log

$rm-rfaudit.log

AddtheauditrulesfortheDockerserviceandaudittheDockerservice:

$nanoaudit.rules

-w/usr/bin/docker-kdocker

$serviceauditdrestart

$ausearch-kdocker

<nomatches>

$dockerps

CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES

$ausearch-kdocker

----

time->FriNov2702:29:502015

type=PROCTITLEmsg=audit(1448620190.716:79):proctitle=646F636B6572007073

type=PATHmsg=audit(1448620190.716:79):item=1name="/lib64/ld-linux-x86-

64.so.2"inode=398512dev=08:01mode=0100755ouid=0ogid=0rdev=00:00

nametype=NORMAL

type=PATHmsg=audit(1448620190.716:79):item=0name="/usr/bin/docker"

inode=941134dev=08:01mode=0100755ouid=0ogid=0rdev=00:00

nametype=NORMAL

https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.6_Benchmark_v1.0.0.pdf

type=CWDmsg=audit(1448620190.716:79):cwd="/etc/audit"

type=EXECVEmsg=audit(1448620190.716:79):argc=2a0="docker"a1="ps"

type=SYSCALLmsg=audit(1448620190.716:79):arch=c000003esyscall=59

success=yesexit=0a0=ca1208a1=c958c8a2=c8

CreateauserforthecontainerCurrently,mappingthecontainer’srootusertoanon-rootuseronthehostisnotsupportedbyDocker.Thesupportforusernamespacewouldbeprovidedinfuturereleases.Thiscreatesaserioususerisolationissue.Itisthushighlyrecommendedtoensurethatthereisanon-rootusercreatedforthecontainerandthecontainerisrunusingthatuser.

Aswecanseeinthefollowingsnippet,bydefault,thecentosDockerimagehasauserfieldasblank,whichmeans,bydefault,thecontainerwillgetarootuserduringruntime,whichshouldbeavoided:

$dockerinspectcentos

[

{

"Id":

"e9fa5d3a0d0e19519e66af2dd8ad6903a7288de0e995b6eafbcb38aebf2b606d",

"RepoTags":[

"centos:latest"

],

"RepoDigests":[],

"Parent":

"c9853740aa059d078b868c4a91a069a0975fb2652e94cc1e237ef9b961afa572",

"Comment":"",

"Created":"2015-10-13T23:29:04.138328589Z",

"Container":

"eaa200e2e187340f0707085b9b4eab5658b13fd190af68c71a60f6283578172f",

"ContainerConfig":{

"Hostname":"7aa5783a47d5",

"Domainname":"",

"User":"",

contd

WhilebuildingtheDockerimage,wecanprovidethetestuser,theless-privilegeduser,intheDockerfile,asshowninthefollowingsnippet:

$cd

$mkdirtest-container

$cdtest-container/

$catDockerfile

FROMcentos:latest

RUNuseraddtest

USERtest

root@ubuntu:~/test-container#dockerbuild-tvkohli.

SendingbuildcontexttoDockerdaemon2.048kB

Step1:FROMcentos:latest

--->e9fa5d3a0d0e

Step2:RUNuseraddtest

--->Runningin0c726d186658

--->12041ebdfd3f

Removingintermediatecontainer0c726d186658

Step3:USERtest

--->Runningin86c5e0599c72

--->af4ba8a0fec5

Removingintermediatecontainer86c5e0599c72

Successfullybuiltaf4ba8a0fec5

$dockerimages|grepvkohli

vkohlilatestaf4ba8a0fec59secondsago172.6MB

WhenwestarttheDockercontainer,wecanseethatitgetsatestuser,andthedockerinspectcommandalsoshowsthedefaultuserastest:

$dockerrun-itvkohli/bin/bash

[test@2ff11ee54c5f/]$whoami

test

[test@2ff11ee54c5f/]$exit

$dockerinspectvkohli

[

{

"Id":

"af4ba8a0fec558d68b4873e2a1a6d8a5ca05797e0bfbab0772bcedced15683ea",

"RepoTags":[

"vkohli:latest"

],

"RepoDigests":[],

"Parent":

"12041ebdfd3f38df3397a8961f82c225bddc56588e348761d3e252eec868d129",

"Comment":"",

"Created":"2015-11-27T14:10:49.206969614Z",

"Container":

"86c5e0599c72285983f3c5511fdec940f70cde171f1bfb53fab08854fe6d7b12",

"ContainerConfig":{

"Hostname":"7aa5783a47d5",

"Domainname":"",

"User":"test",

Contd..

DonotmountsensitivehostsystemdirectoriesoncontainersIfsensitivedirectoriesaremountedinread-writemode,itwouldbepossibletomakechangestofileswithinthosesensitivedirectories.ThechangesmightbringdownsecurityimplicationsorunwarrantedchangesthatcouldputtheDockerhostinacompromisedstate.

Ifthe/run/systemdsensitivedirectoryismountedinthecontainerthenwecanactuallyshutdownthehostfromthecontaineritself:

$dockerrun-ti-v/run/systemd:/run/systemdcentos/bin/bash

[root@1aca7fe47882/]#systemctlstatusdocker

docker.service-DockerApplicationContainerEngine

Loaded:loaded(/lib/systemd/system/docker.service;enabled)

Active:active(running)sinceSun2015-11-2912:22:50UTC;21minago

Docs:https://docs.docker.com

MainPID:758

CGroup:/system.slice/docker.service

[root@1aca7fe47882/]#shutdown

Itcanbeauditedbyusingthefollowingcommand,whichreturnsthelistofcurrentmappeddirectoriesandwhethertheyaremountedinread-writemodeforeachcontainerinstance:

$dockerps-q|xargsdockerinspect--format'{{.Id}}:Volumes={{

.Volumes}}VolumesRW={{.VolumesRW}}'

DonotuseprivilegedcontainersDockersupportstheadditionandremovalofcapabilities,allowingtheuseofanon-defaultprofile.ThismaymakeDockermoresecurethroughcapabilityremoval,orlesssecurethroughtheadditionofcapabilities.Itisthusrecommendedtoremoveallcapabilitiesexceptthoseexplicitlyrequiredforyourcontainerprocess.

Asseeninthefollowing,whenwerunthecontainerwithouttheprivilegedmode,weareunabletochangethekernelparameters,butwhenwerunthecontainerinprivilegedmodeusingthe--privilegedflag,itispossibletochangethekernelparameterseasily,whichcancausesecurityvulnerability:

$dockerrun-itcentos/bin/bash

[root@7e1b1fa4fb89/]#sysctl-wnet.ipv4.ip_forward=0

sysctl:settingkey"net.ipv4.ip_forward":Read-onlyfilesystem

$dockerrun--privileged-itcentos/bin/bash

[root@930aaa93b4e4/]#sysctl-a|wc-l

sysctl:readingkey"net.ipv6.conf.all.stable_secret"

sysctl:readingkey"net.ipv6.conf.default.stable_secret"

sysctl:readingkey"net.ipv6.conf.eth0.stable_secret"

sysctl:readingkey"net.ipv6.conf.lo.stable_secret"

638

[root@930aaa93b4e4/]#sysctl-wnet.ipv4.ip_forward=0

net.ipv4.ip_forward=0

So,whileauditing,itshouldbemadesurethatallthecontainersshouldnothavetheprivilegedmodesettotrue:

$dockerps-q|xargsdockerinspect--format'{{.Id}}:Privileged={{

.HostConfig.Privileged}}'

930aaa93b4e44c0f647b53b3e934ce162fbd9ef1fd4ec82b826f55357f6fdf3a:

Privileged=true

SummaryInthischapter,wetookadeepdiveintoDockersecuritywithanoverviewofcgroupsandkernelnamespace.WealsowentoversomeoftheaspectsoffilesystemsandLinuxcapabilities,whichcontainersleverageinordertoprovidemorefeatures,suchastheprivilegedcontainers,butatthecostofexposingitselfmoreonthethreatside.WealsosawhowcontainerscanbedeployedinasecuredenvironmentinAWSECS(EC2containerservice)usingproxycontainerstorestrictvulnerabletraffic.AppArmoralsoprovideskernel-enhancementfeaturesinordertoconfineapplicationstoalimitedsetofresources.LeveragingtheirbenefitstoDockercontainershelpsustodeploytheminasecuredenvironment.Finally,wehadaquickdiveintoDockersecuritybenchmarksandsomeoftheimportantrecommendationsthatcanbefollowedduringauditingandDockerdeploymentintheproductionenvironment.

Inthenextchapter,wewilllearnabouttuningandtroubleshootingintheDockernetworkusingvarioustools.

Chapter6.NextGenerationNetworkingStackforDocker:libnetworkInthischapter,wewilllearnaboutanewnetworkingstackforDocker:libnetwork,whichprovidesapluggablearchitecturewithadefaultimplementationforsingleandmulti-hostvirtualnetworking:

Introduction

GoalDesign

CNMobjects

CNMattributesCNMlifecycle

Drivers

BridgedriverOverlaynetworkdriver

UsingoverlaynetworkwithVagrantOverlaynetworkwithDockerMachineandDockerSwarmCreatinganoverlaynetworkmanuallyandusingitforcontainersContainernetworkinterfaceCalico’slibnetworkdriver

GoallibnetworkwhichiswritteningolanguageisanewwayforconnectingDockercontainers.Theaimistoprovideacontainernetworkmodelthathelpsprogrammersandprovidestheabstractionofnetworklibraries.Thelong-termgoaloflibnetworkistofollowtheDockerandLinuxphilosophytodelivermodulesthatworkindependently.libnetworkhastheaimtoprovideacomposableneedfornetworkingincontainers.ItalsoaimstomodularizethenetworkinglogicinDockerEngineandlibcontainerintoasingle,reusablelibraryby:

ReplacingthenetworkingmoduleofDockerEnginewithlibnetworkBeingamodelthatallowslocalandremotedriverstoprovidenetworkingtocontainersProvidingatooldnetformanagingandtestinglibnetwork—stillaworkinprogress(referencefromhttps://github.com/docker/libnetwork/issues/45).

https://github.com/docker/libnetwork/issues/45

Designlibnetworkimplementsacontainernetworkmodel(CNM).Itformalizesthestepsrequiredtoprovidenetworkingforcontainers,whileprovidinganabstractionthatcanbeusedtosupportmultiplenetworkdrivers.ItsendpointAPIsareprimarilyusedformanagingthecorrespondingobjectandbook-keepstheminordertoprovidealevelofabstractionasrequiredbytheCNMmodel.

TheCNMisbuiltonthreemaincomponents.Thefollowingfigureshowsthenetworksandboxmodeloflibnetwork:

CNMobjectsLet’sdiscusstheCNMobjectsindetail.

SandboxThiscontainstheconfigurationofacontainer’snetworkstack,whichincludesmanagementofroutingtables,thecontainer’sinterface,andDNSsettings.AnimplementationofasandboxcanbeaLinuxnetworknamespace,aFreeBSDjail,orothersimilarconcept.Asandboxmaycontainmanyendpointsfrommultiplenetworks.Italsorepresentsacontainer’snetworkconfigurationsuchasIP-address,MACaddress,andDNSentries.libnetworkmakesuseoftheOS-specificparameterstopopulatethenetworkconfigurationrepresentedbysandbox.libnetworkprovidesaframeworktoimplementsandboxinmultipleoperatingsystems.Netlinkisusedtomanagetheroutingtableinnamespace,andcurrentlytwoimplementationsofsandboxexist,namespace_linux.goandconfigure_linux.go,touniquelyidentifythepathonthehostfilesystem.

AsandboxisassociatedwithasingleDockercontainer.Thefollowingdatastructureshowstheruntimeelementsofasandbox:

typesandboxstruct{

idstring

containerIDstring

configcontainerConfig

osSboxosl.Sandbox

controller*controller

refCntint

endpointsepHeap

epPrioritymap[string]int

joinLeaveDonechanstruct{}

dbIndexuint64

dbExistsbool

isStubbool

inDeletebool

sync.Mutex

}

Anewsandboxisinstantiatedfromanetworkcontroller(whichisexplainedinmoredetaillater):

func(c*controller)NewSandbox(containerIDstring,options…SandboxOption)

(Sandbox,error){

…..

}

EndpointAnendpointjoinsasandboxtothenetworkandprovidesconnectivityforservicesexposedbyacontainertotheothercontainersdeployedinthesamenetwork.ItcanbeaninternalportofOpenvSwitchorasimilarvethpair.Anendpointcanbelongtoonlyonenetworkbutmayonlybelongtoonesandbox.AnendpointrepresentsaserviceandprovidesvariousAPIstocreateandmanagetheendpoint.Ithasaglobalscopebutgetsattachedtoonlyonenetwork,asshowninthefollowingfigure:

Anendpointisspecifiedbythefollowingdatastructure:

typeendpointstruct{

namestring

idstring

network*network

iface*endpointInterface

joinInfo*endpointJoinInfo

sandboxIDstring

exposedPorts[]types.TransportPort

anonymousbool

genericmap[string]interface{}

joinLeaveDonechanstruct{}

prefAddressnet.IP

prefAddressV6net.IP

ipamOptionsmap[string]string

dbIndexuint64

dbExistsbool

sync.Mutex

}

AnendpointisassociatedwithauniqueIDandname.ItisattachedtoanetworkandasandboxID.ItisalsoassociatedwithanIPv4andIPv6addressspace.EachendpointisassociatedwithanendpointInterfacestruct.

NetworkAnetworkisagroupofendpointsthatareabletocommunicatewitheachotherdirectly.Itprovidestherequiredconnectivitywithinthesamehostormultiplehosts,andwheneveranetworkiscreatedorupdated,thecorrespondingdriverisnotified.AnexampleisaVLANorLinuxbridge,whichhasaglobalscopewithinacluster.

Networksarecontrolledfromanetworkcontroller,whichwewilldiscussinthenextsection.Everynetworkhasaname,addressspace,ID,andnetworktype:

typenetworkstruct{

ctrlr*controller

namestring

networkTypestring

idstring

ipamTypestring

addrSpacestring

ipamV4Config[]*IpamConf

ipamV6Config[]*IpamConf

ipamV4Info[]*IpamInfo

ipamV6Info[]*IpamInfo

enableIPv6bool

postIPv6bool

epCnt*endpointCnt

genericoptions.Generic

dbIndexuint64

svcRecordssvcMap

dbExistsbool

persistbool

stopWatchChchanstruct{}

drvOnce*sync.Once

internalbool

sync.Mutex

}

NetworkcontrollerAnetworkcontrollerobjectprovidesAPIstocreateandmanageanetworkobject.Itisanentrypointinthelibnetworkbybindingaparticulardrivertoagivennetwork,anditsupportsmultipleactivedrivers,bothin-builtandremote.Networkcontrollerallowsuserstobindaparticulardrivertoagivennetwork:

typecontrollerstruct{

idstring

driversdriverTable

ipamDriversipamTable

sandboxessandboxTable

cfg*config.Config

stores[]datastore.DataStore

discoveryhostdiscovery.HostDiscovery

extKeyListenernet.Listener

watchChchan*endpoint

unWatchChchan*endpoint

svcDbmap[string]svcMap

nmapmap[string]*netWatch

defOsSboxosl.Sandbox

sboxOncesync.Once

sync.Mutex

}

Eachnetworkcontrollerhasreferencetothefollowing:

OneormoredriversinthedatastructuredriverTableOneormoresandboxesinthedatastructureDataStoreipamTable

ThefollowingfigureshowshowNetworkControllersitsbetweentheDockerEngineandthecontainersandnetworkstheyareattachedto:

CNMattributesTherearetwotypesofattributes,asfollows:

Options:Theyarenotend-uservisiblebutarethekey-valuepairsofdatatoprovideaflexiblemechanismtopassdriver-specificconfigurationfromusertodriverdirectly.libnetworkoperatesontheoptionsonlyifthekeymatchesawell-knownlabelasaresultvalueispickedup,whichisrepresentedbyagenericobject.Labels:Theyareasubsetofoptionsthatareend-uservariablesrepresentedintheUIusingthe–labelsoption.Theirmainfunctionistoperformdriver-specificoperationsandtheyarepassedfromtheUI.

CNMlifecycleConsumersofthecontainernetworkmodelinteractthroughtheCNMobjectsanditsAPIstonetworkthecontainersthattheymanage.

Driversregisterwithnetworkcontroller.Built-indriversregisterinsideoflibnetwork,whileremotedriversregisterwithlibnetworkviaapluginmechanism(WIP).Eachdriverhandlesaparticularnetworktype.

Anetworkcontrollerobjectiscreatedusingthelibnetwork.New()APItomanagetheallocationofnetworksandoptionallyconfigureadriverwithdriver-specificoptions.

Thenetworkiscreatedusingthecontroller’sNewNetwork()APIbyprovidinganameandnetworkType.ThenetworkTypeparameterhelpstochooseacorrespondingdriverandbindsthecreatednetworktothatdriver.Fromthispoint,anyoperationonthenetworkwillbehandledbythatdriver.

Thecontroller.NewNetwork()APIalsotakesinoptionaloptionsparametersthatcarrydriver-specificoptionsandlabels,whichthedriverscanmakeuseforitspurpose.

network.CreateEndpoint()canbecalledtocreateanewendpointinagivennetwork.ThisAPIalsoacceptsoptionaloptionsparametersthatvarywiththedriver.

Driverswillbecalledwithdriver.CreateEndpointanditcanchoosetoreserveIPv4/IPv6addresseswhenanendpointiscreatedinanetwork.ThedriverwillassigntheseaddressesusingtheInterfaceInfointerfacedefinedinthedriverAPI.TheIPv4/IPv6addressesareneededtocompletetheendpointasaservicedefinitionalongwiththeportstheendpointexposes.Aserviceendpointisanetworkaddressandtheportnumberthattheapplicationcontainerislisteningon.

endpoint.Join()canbeusedtoattachacontainertoanendpoint.TheJoinoperationwillcreateasandboxifitdoesn’texistforthatcontainer.Thedriversmakeuseofthesandboxkeytoidentifymultipleendpointsattachedtothesamecontainer.

ThereisaseparateAPItocreateanendpointandanothertojointheendpoint.

Anendpointrepresentsaservicethatisindependentofthecontainer.Whenanendpointiscreated,ithasresourcesreservedforthecontainertogetattachedtotheendpointlater.Itgivesaconsistentnetworkingbehavior.

endpoint.Leave()isinvokedwhenacontainerisstopped.ThedrivercancleanupthestatesthatitallocatedduringtheJoin()call.libnetworkwilldeletethesandboxwhenthelastreferencingendpointleavesthenetwork.

libnetworkkeepsholdingontoIPaddressesaslongastheendpointisstillpresent.Thesewillbereusedwhenthecontainer(oranycontainer)joinsagain.Itensuresthatthecontainer’sresourcesarere-usedwhentheyarestoppedandstartedagain.

endpoint.Delete()isusedtodeleteanendpointfromanetwork.Thisresultsindeletingtheendpointandcleaningupthecachedsandbox.Info.

network.Delete()isusedtodeleteanetwork.Deleteisallowediftherearenoendpoints

attachedtothenetwork.

DriverAdriverownsanetworkandisresponsibleformakingthenetworkworkandmanagesit.NetworkcontrollerprovidesanAPItoconfigurethedriverwithspecificlabels/optionsthatarenotdirectlyvisibletotheuserbutaretransparenttolibnetworkandcanbehandledbydriversdirectly.Driverscanbebothin-built(suchasbridge,host,oroverlay)andremote(frompluginproviders)tobedeployedinvarioususecasesanddeploymentscenarios.

Thedriverownsthenetworkimplementationandisresponsibleformanagingit,includingIPAddressManagement(IPAM).Thefollowingfigureexplainstheprocess:

Thefollowingarethein-builtdrivers:

Null:Inordertoprovidebackwardcompatibilitywitholddocker--net=none,thisoptionexistsprimarilyinthecasewhennonetworkingisrequired.Bridge:ItprovidesaLinux-specificbridgingimplementationdriver.Overlay:TheoverlaydriverimplementsnetworkingthatcanspanmultiplehostsnetworkencapsulationsuchasVXLAN.Wewillbedoingadeep-diveontwoofitsimplementations:basicsetupwithConsulandVagrantsetuptodeploytheoverlaydriver.Remote:Itprovidesameansofsupportingdriversoveraremotetransportandaspecificdrivercanbewrittenasperchoice.

BridgedriverAbridgedriverrepresentsawrapperonaLinuxbridgeactingasanetworkforlibcontainer.Itcreatesavethpairforeachnetworkcreated.Oneendisconnectedtothecontainerandtheotherendisconnectedtothebridge.Thefollowingdatastructurerepresentsabridgenetwork:

typedriverstruct{

config*configuration

etwork*bridgeNetwork

natChain*iptables.ChainInfo

filterChain*iptables.ChainInfo

networksmap[string]*bridgeNetwork

storedatastore.DataStore

sync.Mutex

}

Someoftheactionsperformedinabridgedriver:

ConfiguringIPTablesManagingIPforwardingManagingPortMappingEnablingBridgeNetFilteringSettingupIPv4andIPv6onthebridge

Thefollowingdiagramshowshowthenetworkisrepresentedusingdocker0andvethpairstoconnectendpointswiththedocker0bridge:

OverlaynetworkdriverOverlaynetworkinlibnetworkusesVXLanalongwithaLinuxbridgetocreateanoverlaidaddressspace.Itsupportsmulti-hostnetworking:

const(

networkType="overlay"

vethPrefix="veth"

vethLen=7

vxlanIDStart=256

vxlanIDEnd=1000

vxlanPort=4789

vxlanVethMTU=1450

)

typedriverstruct{

eventChchanserf.Event

notifyChchanovNotify

exitChchanchanstruct{}

bindAddressstring

neighIPstring

configmap[string]interface{}

peerDbpeerNetworkMap

serfInstance*serf.Serf

networksnetworkTable

storedatastore.DataStore

ipAllocator*idm.Idm

vxlanIdm*idm.Idm

oncesync.Once

joinOncesync.Once

sync.Mutex

}

UsingoverlaynetworkwithVagrantOverlaynetworkiscreatedbetweentwocontainers,andVXLantunnelconnectsthecontainersthroughabridge.

OverlaynetworkdeploymentVagrantsetupThissetuphasbeendeployedusingtheDockerexperimentalversion,whichkeepsonupdatingregularlyandmightnotsupportsomeofthefeatures:

1. Clonetheofficiallibnetworkrepositoryandswitchtothedocsfolder:

$gitclone

$cd

libnetwork/docs

2. TheVagrantscriptpre-existsintherepository;wewilldeploythethree-nodesetupforourDockeroverlaynetworkdrivertestingbyusingthefollowingcommand:

$vagrantup

Bringingmachine'consul-server'upwith'virtualbox'provider…

Bringingmachine'net-1'upwith'virtualbox'provider…

Bringingmachine'net-2'upwith'virtualbox'provider…

==>consul-server:Box'ubuntu/trusty64'couldnotbefound.

Attemptingtofindandinstall…

consul-server:BoxProvider:virtualbox

consul-server:BoxVersion:>=0

==>consul-server:Loadingmetadataforbox'ubuntu/trusty64'

consul-server:URL:https://atlas.hashicorp.com/ubuntu/trusty64

==>consul-server:Addingbox'ubuntu/trusty64'(v20151217.0.0)for

provider:virtualbox

consul-server:Downloading:

https://atlas.hashicorp.com/ubuntu/boxes/trusty64/versions/20151217.0.0

/providers/virtualbox.box

==>consul-server:Successfullyaddedbox'ubuntu/trusty64'

(v20151217.0.0)for'virtualbox'!

==>consul-server:Importingbasebox'ubuntu/trusty64'...

==>consul-server:MatchingMACaddressforNATnetworking…

==>consul-server:Checkingifbox'ubuntu/trusty64'isuptodate…

==>consul-server:SettingthenameoftheVM:

libnetwork_consul-server_1451244524836_56275

==>consul-server:Clearinganypreviouslysetforwardedports…

==>consul-server:Clearinganypreviouslysetnetworkinterfaces…

==>consul-server:Preparingnetworkinterfacesbasedon

configuration…

consul-server:Adapter1:nat

consul-server:Adapter2:hostonly

==>consul-server:Forwardingports…

consul-server:22=>2222(adapter1)

==>consul-server:Running'pre-boot'VMcustomizations…

==>consul-server:BootingVM…

==>consul-server:Waitingformachinetoboot.Thismaytakeafew

minutes…

consul-server:

101aac79c475b84f6aff48352ead467d6b2b63ba6b64cc1b93c630489f7e3f4c

==>net-1:Box'ubuntu/vivid64'couldnotbefound.Attemptingtofind

andinstall…

net-1:BoxProvider:virtualbox

net-1:BoxVersion:>=0

==>net-1:Loadingmetadataforbox'ubuntu/vivid64'

net-1:URL:https://atlas.hashicorp.com/ubuntu/vivid64

\==>net-1:Addingbox'ubuntu/vivid64'(v20151219.0.0)forprovider:

virtualbox

net-1:Downloading:

https://atlas.hashicorp.com/ubuntu/boxes/vivid64/versions/20151219.0.0/

providers/virtualbox.box

contd…

3. WecanlistthedeployedmachinebyVagrantasfollows:

$vagrantstatus

Currentmachinestates:

consul-serverrunning(virtualbox)

net-1running(virtualbox)

net-2running(virtualbox)

ThisenvironmentrepresentsmultipleVMs.TheVMsarealllistedabove

withtheircurrentstate.FormoreinformationaboutaspecificVM,run

`vagrantstatusNAME`.

4. ThesetupiscompletethankstotheVagrantscript;now,wecanSSHtotheDockerhostsandstartthetestingcontainers:

$vagrantsshnet-1

WelcometoUbuntu15.04(GNU/Linux3.19.0-42-genericx86_64)

*Documentation:https://help.ubuntu.com/

SysteminformationasofSunDec2720:04:06UTC2015

Systemload:0.0Usersloggedin:0

Usageof/:4.5%of38.80GBIPaddressforeth0:10.0.2.15

Memoryusage:24%IPaddressforeth1:192.168.33.11

Swapusage:0%IPaddressfordocker0:172.17.0.1

Processes:78

Graphthisdataandmanagethissystemat:

https://landscape.canonical.com/

GetcloudsupportwithUbuntuAdvantageCloudGuest:

http://www.ubuntu.com/business/services/cloud

5. WecancreateanewDockercontainer,andinsidethecontainerwecanlistthecontentsofthe/etc/hostsfileinordertoverifythatithastheoverlaybridgespecification,whichwaspreviouslydeployed,anditautomaticallyconnectstoitonthelaunch:

$dockerrun-it--rmubuntu:14.04bash

Unabletofindimage'ubuntu:14.04'locally

14.04:Pullingfromlibrary/ubuntu

6edcc89ed412:Pullcomplete

bdf37643ee24:Pullcomplete

ea0211d47051:Pullcomplete

a3ed95caeb02:Pullcomplete

Digest:

sha256:d3b59c1d15c3cfb58d9f2eaab8a232f21fc670c67c11f582bc48fb32df17f3b3

Status:Downloadednewerimageforubuntu:14.04

root@65db9144c65b:/#cat/etc/hosts

172.21.0.42ac726b4ce60

127.0.0.1localhost

::1localhostip6-localhostip6-loopback

fe00::0ip6-localnet

ff00::0ip6-mcastprefix

ff02::1ip6-allnodes

ff02::2ip6-allrouters

172.21.0.3distracted_bohr

172.21.0.3distracted_bohr.multihost

172.21.0.4modest_curie

172.21.0.4modest_curie.multihost

6. Similarly,wecancreatetheDockercontainerintheotherhostnet-2aswellandcanverifytheworkingoftheoverlaynetworkdriverasboththecontainerswillbeabletopingeachotherinspiteofbeingdeployedondifferenthosts.

Inthepreviousexample,westartedtheDockercontainerwiththedefaultoptionsandtheygotautomaticallyaddedtoamulti-hostnetworkoftypeoverlay.

Wecanalsocreataseparateoverlaybridgeandaddcontainerstoitmanuallyusingthe--publish-serviceoption,whichispartofDockerexperimental:

vagrant@net-1:~$dockernetworkcreate-doverlaytester

447e75fd19b236e72361c270b0af4402c80e1f170938fb22183758c444966427

vagrant@net-1:~$dockernetworkls

NETWORKIDNAMEDRIVE

447e75fd19b2testeroverlay

b77a7d741b45bridgebridge

40fe7cfeee20nonenull

62072090b6achosthost

ThesecondhostwillalsoseethisnetworkandwecancreatecontainersaddedtotheoverlaynetworkinbothofthesehostsbyusingthefollowingoptionintheDockercommand:

$dockerrun-it--rm--publish-service=bar.tester.overlayubuntu:14.04

bash

Wewillbeabletoverifytheworkingoftheoverlaydriverasboththecontainerswillbeabletopingeachother.Also,toolssuchastcpdump,wireshark,smartsniff,andsooncanbeusedtocapturethevXLANpackage.

OverlaynetworkwithDockerMachineandDockerSwarmThissectionexplainsthebasicsofcreatingamulti-hostnetwork.TheDockerEnginesupportsmulti-hostnetworkingthroughtheoverlaynetworkdriver.Overlaydriversneedthefollowingpre-requisitestowork:

3.16LinuxkernelorhigherAccesstoakey-valuestoreDockersupportsthefollowingkey-valuestores:Consul,etcd,andZooKeeperAclusterofhostsconnectedtothekey-valuestoreDockerEnginedaemononeachhostinthecluster

ThisexampleusesDockerMachineandDockerSwarmtocreatethemulti-networkhost.

DockerMachineisusedtocreatethekey-valuestoreserverandthecluster.TheclustercreatedisaDockerSwarmcluster.

ThefollowingdiagramexplainshowthreeVMsaresetupusingDockerMachine:

PrerequisitesVagrantDockerEngineDockerMachineDockerSwarm

Key-valuestoreinstallationAnoverlaynetworkrequiresakey-valuestore.Thekey-valuestorestoresinformationaboutthenetworkstatesuchasdiscovery,networks,endpoints,IPaddresses,andsoon.Dockersupportsvariouskey-valuestoressuchasConsul,etcd,andZooKeeper.ThissectionhasbeenimplementedusingConsul.

Thefollowingarethestepstoinstallkey-valuestore:

1. ProvisionaVirtualBoxvirtualmachinecalledmh-keystore.

WhenanewVMisprovisioned,theprocessaddstheDockerEnginetothehost.ConsulinstancewillbeusingtheconsulimagefromtheDockerHubaccount(https://hub.docker.com/r/progrium/consul/):

$docker-machinecreate-dvirtualboxmh-keystore


Creatingmachine…

(mh-keystore)CreatingVirtualBoxVM…

(mh-keystore)CreatingSSHkey…

(mh-keystore)StartingVM…




Detectingtheprovisioner…

Provisioningwithboot2docker…




CheckingconnectiontoDocker…

Dockerisupandrunning!


mh-keystore

2. Starttheprogrium/consulcontainercreatedpreviouslyrunningonthemh-keystorevirtualmachine:

$docker$(docker-machineconfigmh-keystore)run-d\

>-p"8500:8500"\

>-h"consul"\

>progrium/consul-server–bootstrap

Unabletofindimage'progrium/consul:latest'locally

latest:Pullingfromprogrium/consul

3b4d28ce80e4:Pullcomplete

…

d9125e9e799b:Pullcomplete

Digest:

sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274

Status:Downloadednewerimageforprogrium/consul:latest

032884c7834ce22707ed08068c24c503d599499f1a0a58098c31be9cc84d8e6c

Abashexpansion$(docker-machineconfigmh-keystore)isusedtopasstheconnectionconfigurationtotheDockerruncommand.Theclientstartsaprogram

https://hub.docker.com/r/progrium/consul/

fromtheprogrium/consulimagerunninginthemh-keystoremachine.Thecontaineriscalledconsul(flag–h)andislisteningonport8500(youcanchooseanyotherportaswell).

3. Setthelocalenvironmenttothemh-keystorevirtualmachine:

$eval"$(docker-machineenvmh-keystore)"

4. ExecutethedockerpscommandtomakesuretheConsulcontainerisup:

$dockerps

CONTAINERIDIMAGECOMMANDCREATED

032884c7834cprogrium/consul"/bin/start-server-"47seconds

ago

STATUSPORTS

Up46seconds53/tcp,53/udp,8300-8302/tcp,8301-8302/udp,8400/tcp,

0.0.0.0:8500->8500/tcp

NAMES

sleepy_austin

CreateaSwarmclusterwithtwonodesInthisstep,wewilluseDockerMachinetoprovisiontwohostsforyournetwork.WewillcreatetwovirtualmachinesinVirtualBox.OneofthemachineswillbeSwarmmaster,whichwillbecreatedfirst.

Aseachhostiscreated,optionsfortheoverlaynetworkdriverwillbepassedtotheDockerEngineusingSwarmusingthefollowingsteps:

1. CreateaSwarmmastervirtualmachinemhs-demo0:

$docker-machinecreate\

-dvirtualbox\

--swarm--swarm-master\

--swarm-discovery="consul://$(docker-machineipmh-keystore):8500"\

--engine-opt="cluster-store=consul://$(docker-machineipmh-

keystore):8500"\

--engine-opt="cluster-advertise=eth1:2376"\

mhs-demo0

Atcreationtime,yousupplytheenginedaemonwiththe--cluster-storeoption.Thisoptiontellstheenginethelocationofthekey-valuestorefortheoverlaynetwork.Thebashexpansion$(docker-machineipmh-keystore)resolvestotheIPaddressoftheConsulserveryoucreatedinstep1oftheprecedingsection.The--cluster-advertiseoptionadvertisesthemachineonthenetwork.

2. Createanothervirtualmachinemhs-demo1andaddittotheDockerSwarmcluster:

$docker-machinecreate-dvirtualbox\

--swarm\

--swarm-discovery="consul://$(docker-machineipmh-keystore):8500"

\

--engine-opt="cluster-store=consul://$(docker-machineipmh-

keystore):8500"\

--engine-opt="cluster-advertise=eth1:2376"\

mhs-demo1


Creatingmachine…

(mhs-demo1)CreatingVirtualBoxVM…

(mhs-demo1)CreatingSSHkey…

(mhs-demo1)StartingVM…




Detectingtheprovisioner…

Provisioningwithboot2docker…




Configuringswarm…

CheckingconnectiontoDocker…

Dockerisupandrunning!


mhs-demo1

3. ListvirtualmachinesusingDockerMachinetoconfirmthattheyareallupandrunning:

$docker-machinels

NAMEACTIVEDRIVERSTATEURL

SWARMDOCKERERRORS

mh-keystore*virtualboxRunningtcp://192.168.99.100:2376

v1.9.1

mhs-demo0-virtualboxRunningtcp://192.168.99.101:2376

mhs-demo0(master)v1.9.1

mhs-demo1-virtualboxRunningtcp://192.168.99.102:2376

mhs-demo0v1.9.1

Atthispoint,virtualmachinesarerunning.Wearereadytocreateamulti-hostnetworkforcontainersusingthesevirtualmachines.

CreatinganoverlaynetworkThefollowingcommandisusedtocreateanoverlaynetwork:

$dockernetworkcreate--driveroverlaymy-net

WewillonlyneedtocreatethenetworkonasinglehostintheSwarmcluster.WeusedtheSwarmmasterbutthiscommandcanrunonanyhostintheSwarmcluster:

1. Checkthattheoverlaynetworkisrunningusingthefollowingcommand:

$dockernetworkls

bd85c87911491d7112739e6cf08d732eb2a2841c6ca1efcc04d0b20bbb832a33

rdua1-ltm:overlay-tutorialrdua$dockernetworkls

NETWORKIDNAMEDRIVER

bd85c8791149my-netoverlay

fff23086faa8mhs-demo0/bridgebridge

03dd288a8adbmhs-demo0/nonenull

2a706780454fmhs-demo0/hosthost

f6152664c40amhs-demo1/bridgebridge

ac546be9c37cmhs-demo1/nonenull

c6a2de6ba6c9mhs-demo1/hosthost

SinceweareusingtheSwarmmasterenvironment,weareabletoseeallthenetworksonalltheSwarmagents:thedefaultnetworksoneachengineandthesingleoverlaynetwork.Inthiscase,therearetwoenginesrunningonmhs-demo0andmhs-demo1.

EachNETWORKIDisunique.

2. SwitchtoeachSwarmagentinturnandlistthenetworks:

$eval$(docker-machineenvmhs-demo0)

$dockernetworkls

NETWORKIDNAMEDRIVER


03dd288a8adbnonenull

2a706780454fhosthost

fff23086faa8bridgebridge


$dockernetworkls

NETWORKIDNAMEDRIVER


358c45b96bebdocker_gwbridgebridge

f6152664c40abridgebridge

ac546be9c37cnonenull

c6a2de6ba6c9hosthost

Bothagentsreporttheyhavethemy-netnetworkwiththeoverlaydriver.Wehaveamulti-hostoverlaynetworkrunning.

Thefollowingfigureshowshowtwocontainerswillhavecontainerscreatedandtied

togetherusingtheoverlaymy-net:

CreatingcontainersusinganoverlaynetworkThefollowingarethestepsforcreatingcontainersusinganoverlaynetwork:

1. Createacontainerc0onmhs-demo0andconnecttothemy-netnetwork:


root@843b16be1ae1:/#

$sudodockerrun-i-t--name=c0--net=my-netdebian/bin/bash

ExecuteifconfigtofindtheIPaddressofc0.Inthiscase,itis10.0.0.4:

root@843b16be1ae1:/#ifconfig

eth0Linkencap:EthernetHWaddr02:42:0a:00:00:04


inet6addr:fe80::42:aff:fe00:4/64Scope:Link















inetaddr:127.0.0.1Mask:255.0.0.0







2. Createacontainer,c1onmhs-demo1,andconnecttothemy-netnetwork:


$sudodockerrun-i-t--name=c1--net=my-netdebian/bin/bash

Unabletofindimage'ubuntu:latest'locally

latest:Pullingfromlibrary/ubuntu

0bf056161913:Pullcomplete

1796d1c62d0c:Pullcomplete

e24428725dd6:Pullcomplete

89d5d8e8bafb:Pullcomplete

Digest:

sha256:a2b67b6107aa640044c25a03b9e06e2a2d48c95be6ac17fb1a387e75eebafd7c

Status:Downloadednewerimageforubuntu:latest

root@2ce83e872408:/#

3. ExecuteifconfigtofindtheIPaddressofc1.Inthiscase,itis10.0.0.3:

root@2ce83e872408:/#ifconfig

eth0Linkencap:EthernetHWaddr02:42:0a:00:00:03


inet6addr:fe80::42:aff:fe00:3/64Scope:Link















inetaddr:127.0.0.1Mask:255.0.0.0







4. Pingc1(10.0.0.3)fromc0(10.0.0.4)andviceversa:

root@2ce83e872408:/#ping10.0.04

PING10.0.04(10.0.0.4)56(84)bytesofdata.




ContainernetworkinterfaceContainernetworkinterface(CNI)isaspecificationthatdefineshowexecutablepluginscanbeusedtoconfigurenetworkinterfacesforLinuxapplicationcontainers.TheofficialGitHubrepositoryofCNIexplainshowagolibraryexplainstheimplementingspecification.

Thecontainerruntimefirstcreatesanewnetworknamespaceforthecontainerinwhichitdetermineswhichnetworkthiscontainershouldbelongtoandwhichpluginstobeexecuted.ThenetworkconfigurationisintheJSONformatanddefinesonthecontainerstartupwhichpluginshouldbeexecutedforthenetwork.CNIisactuallyanevolvingopensourcetechnologythatisderivedfromtherktnetworkingprotocol.EachCNIpluginisimplementedasanexecutableandisinvokedbyacontainermanagementsystem,docker,orrkt.

Afterinsertingthecontainerinthenetworknamespace,namelybyattachingoneendofavethpairtoacontainerandattachingtheotherendtoabridge,itthenassignsanIPtotheinterfaceandsetsuproutesconsistentwithIPaddressmanagementbyinvokinganappropriateIPAMplugin.

TheCNImodeliscurrentlyusedforthenetworkingofkubeletsintheKubernetesmodel.KubeletsarethemostimportantcomponentsofKubernetesnodes,whichtakestheloadofrunningcontainersontopofthem.

ThepackageCNIforkubeletisdefinedinthefollowingKubernetespackage:

Constants

const(

CNIPluginName="cni"

DefaultNetDir="/etc/cni/net.d"

DefaultCNIDir="/opt/cni/bin"

DefaultInterfaceName="eth0"

VendorCNIDirTemplate="%s/opt/%s/bin"

)

funcProbeNetworkPlugins

funcProbeNetworkPlugins(pluginDirstring)[]network.NetworkPlugin

ThefollowingfigureshowstheCNIplacement:

CNIpluginAspertheofficialGitHubrepository(https://github.com/appc/cni),theparametersthattheCNIpluginneedinordertoaddacontainertothenetworkare:

Version:TheversionofCNIspecthatthecallerisusing(containercallinvokingtheplugin).ContainerID:Thisisoptional,butrecommended,anddefinesthatthereshouldbeauniqueIDacrossanadministrativedomainwhilethecontainerislive.Forexample,theIPAMsystemmayrequirethateachcontainerisallocatedauniqueIDsothatitcanbecorrelatedproperlytoacontainerrunninginthebackground.Networknamespacepath:Thisrepresentsthepathtothenetworknamespacetobeadded,forexample,/proc/[pid]/ns/netorabind-mount/linktoit.Networkconfiguration:ItistheJSONdocumentthatdescribesanetworktowhichacontainercanbejoinedandisexplainedinthefollowingsection.Extraarguments:ItallowsgranularconfigurationofCNIpluginsonaper-containerbasis.Nameoftheinterfaceinsidethecontainer:ItisthenamethatgetsassignedtothecontainerandcomplieswithLinuxrestriction,whichexistsforinterfacenames.

Theresultsachievedareasfollows:

IPsassignedtotheinterface:ThisiseitheranIPv4addressoranIPv6addressassignedtothenetworkasperrequirements.ListofDNSnameservers:Thisisapriority-orderedaddresslistofDNSnameservers.

https://github.com/appc/cni

NetworkconfigurationThenetworkconfigurationisintheJSONformatthatcanbestoredondiskorgeneratedfromothersourcesbycontainerruntime.ThefollowingfieldsintheJSONhaveimportance,asexplainedinthefollowing:

cniVersion(string):ItisSemanticVersion2.0oftheCNIspecificationtowhichthisconfigurationmeets.name(string):Itisthenetworkname.Itisuniqueacrossallcontainersonthehost(orotheradministrativedomain).type(string):ReferstothefilenameoftheCNIpluginexecutable.ipMasq(boolean):Optional,setsupanIPmasqueradeonthehostasitisnecessaryforthehosttoactasagatewaytosubnetsthatarenotabletoroutetotheIPassignedtothecontainer.ipam:DictionarywithIPAM-specificvalues.type(string):ReferstothefilenameoftheIPAMpluginexecutable.routes(list):Listofsubnets(inCIDRnotation)thattheCNIpluginshouldmakesurearereachablebyroutingthroughthenetwork.Eachentryisadictionarycontaining:

dst(string):AsubnetinCIDRnotationgw(string):ItistheIPaddressofthegatewaytouse.Ifnotspecified,thedefaultgatewayforthesubnetisassumed(asdeterminedbytheIPAMplugin).

Anexampleconfigurationforplugin-specificOVSisasfollows:

{

"cniVersion":"0.1.0",

"name":"pci",

"type":"ovs",

//type(plugin)specific

"bridge":"ovs0",

"vxlanID":42,

"ipam":{

"type":"dhcp",

"routes":[{"dst":"10.3.0.0/16"},{"dst":"10.4.0.0/16"}]

}

}

IPallocationTheCNIpluginassignsanIPaddresstotheinterfaceandinstallsnecessaryroutesfortheinterface,thusitprovidesgreatflexibilityfortheCNIpluginandmanyCNIpluginsinternallyhavethesamecodetosupportseveralIPmanagementschemes.

TolessentheburdenontheCNIplugin,asecondtypeofplugin,IPaddressmanagementplugin(IPAM),isdefined,whichdeterminestheinterfaceIP/subnet,gateway,androutesandreturnsthisinformationtothemainplugintoapply.TheIPAMpluginobtainsinformationviaaprotocol,ipamsectiondefinedinthenetworkconfigurationfile,ordatastoredonthelocalfilesystem.

IPaddressmanagementinterfaceTheIPAMpluginisinvokedbyrunninganexecutable,whichissearchedinapredefinedpathandisindicatedbyaCNIpluginviaCNI_PATH.TheIPAMpluginreceivesallthesystemenvironmentvariablesfromthisexecutable,whicharepassedtotheCNIplugin.

IPAMreceivesanetworkconfigurationfileviastdin.SuccessisindicatedbyazeroreturncodeandthefollowingJSON,whichgetsprintedtostdout(inthecaseoftheADDcommand):

{

"cniVersion":"0.1.0",

"ip4":{

"ip":<ipv4-and-subnet-in-CIDR>,

"gateway":<ipv4-of-the-gateway>,(optional)

"routes":<list-of-ipv4-routes>(optional)

},

"ip6":{

"ip":<ipv6-and-subnet-in-CIDR>,

"gateway":<ipv6-of-the-gateway>,(optional)

"routes":<list-of-ipv6-routes>(optional)

},

"dns":<list-of-DNS-nameservers>(optional)

}

ThefollowingisanexampleofrunningDockernetworkingwithCNI:

1. First,installGoLang1.4+andjq(commandlineJSONprocessor)tobuildtheCNIplugins:

$wgethttps://storage.googleapis.com/golang/go1.5.2.linux-amd64.tar.gz

$tar-C/usr/local-xzfgo1.5.2.linux-amd64.tar.gz

$exportPATH=$PATH:/usr/local/go/bin

$goversion

goversiongo1.5.2linux/amd64

$sudoapt-getinstalljq

2. ClonetheofficialCNIGitHubrepository:

$gitclonehttps://github.com/appc/cni.git

Cloninginto'cni'...

remote:Countingobjects:881,done.

remote:Total881(delta0),reused0(delta0),pack-reused881

Receivingobjects:100%(881/881),543.54KiB|313.00KiB/s,done.

Resolvingdeltas:100%(373/373),done.

Checkingconnectivity…done.

3. Wewillnowcreateanetconffileinordertodescribethenetwork:

mkdir-p/etc/cni/net.d

root@rajdeepd-virtual-machine:~#cat>/etc/cni/net.d/10-mynet.conf

<<EOF

>{

>"name":"mynet",

>"type":"bridge",

>"bridge":"cni0",

>"isGateway":true,

>"ipMasq":true,

>"ipam":{

>"type":"host-local",

>"subnet":"10.22.0.0/16",

>"routes":[

>{"dst":"0.0.0.0/0"}

>]

>}

>}

>EOF

4. BuildtheCNIplugins:

~/cni$./build

BuildingAPI

BuildingreferenceCLI

Buildingplugins

flannel

bridge

ipvlan

macvlan

ptp

dhcp

host-local

5. Nowwewillexecutethepriv-net-run.shscriptinordertocreatetheprivatenetworkwiththeCNIplugin:

~/cni/scripts$sudoCNI_PATH=$CNI_PATH./priv-net-run.shifconfig

eth0Linkencap:EthernetHWaddr8a:72:75:7d:6d:6c


inet6addr:fe80::8872:75ff:fe7d:6d6c/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







6. RunaDockercontainerwiththenetworknamespace,whichwassetuppreviouslyusingtheCNIplugin:

~/cni/scripts$sudoCNI_PATH=$CNI_PATH./docker-run.sh--rm

busybox:latest/bin/ifconfig

eth0Linkencap:EthernetHWaddr92:B2:D3:E5:BA:9B


inet6addr:fe80::90b2:d3ff:fee5:ba9b/64Scope:Link







inetaddr:127.0.0.1Mask:255.0.0.0







ProjectCalico’slibnetworkdriverCalicoprovidesascalablenetworkingsolutionforconnectingcontainers,VMs,orbaremetal.CalicoprovidesconnectivityusingthescalableIPnetworkingprincipleasalayer3approach.Calicocanbedeployedwithoutoverlaysorencapsulation.TheCalicoserviceshouldbedeployedasacontaineroneachnodeandprovideseachcontainerwithitsownIPaddress.ItalsohandlesallthenecessaryIProuting,securitypolicyrules,anddistributionofroutesacrossaclusterofnodes.

TheCalicoarchitecturecontainsfourimportantcomponentsinordertoprovideabetternetworkingsolution:

Felix,theCalicoworkerprocess,istheheartofCaliconetworking,whichprimarilyroutesandprovidesdesiredconnectivitytoandfromtheworkloadsonhost.Italsoprovidestheinterfacetokernelsforoutgoingendpointtraffic.BIRD,theroutedistributionopensourceBGP,exchangesroutinginformationbetweenhosts.Thekernelendpoints,whicharepickedupbyBIRD,aredistributedtoBGPpeersinordertoprovideinter-hostrouting.TwoBIRDprocessesruninthecalico-nodecontainer,IPv4(bird)andoneforIPv6(bird6).Confd,atemplatingprocesstoauto-generateconfigurationforBIRD,monitorstheetcdstoreforanychangestoBGPconfigurationsuchasloglevelsandIPAMinformation.ConfdalsodynamicallygeneratesBIRDconfigurationfilesbasedondatafrometcdandtriggersautomaticallyasupdatesareappliedtodata.ConfdtriggersBIRDtoloadnewfileswheneveraconfigurationfileischanged.calicoctl,thecommandlineusedtoconfigureandstarttheCalicoservice,evenallowsthedatastore(etcd)todefineandapplysecuritypolicy.ThetoolalsoprovidesthesimpleinterfaceforgeneralmanagementofCalicoconfigurationirrespectiveofwhetherCalicoisrunningonVMs,containers,orbaremetal.Thefollowingcommandsaresupportedatcalicoctl:

$calicoctlOverridethehost:portoftheETCDserverbysettingthe

environmentvariableETCD_AUTHORITY[default:127.0.0.1:2379]Usage:

calicoctl<command>[<args>...]

statusPrintcurrentstatusinformation

nodeConfigurethemaincalico/nodecontainerand

establishCaliconetworking

containerConfigurecontainersandtheiraddresses

profileConfigureendpointprofiles

endpointConfiguretheendpointsassignedtoexisting

containers

poolConfigureip-pools

bgpConfigureglobalbgp

ipamConfigureIPaddressmanagement

checksystemCheckforincompatibilitiesonthehostsystem

diagsSavediagnosticinformation

versionDisplaytheversionofcalicoctl

configConfigurelow-levelcomponentconfiguration

See'calicoctl<command>--help'toreadaboutaspecificsubcommand.

AspertheofficialGitHubpageoftheCalicorepository

(https://github.com/projectcalico/calico-containers),thefollowingintegrationofCalicoexists:

CalicoasaDockernetworkpluginCalicowithoutDockernetworkingCalicowithKubernetesCalicowithMesosCalicowithDockerSwarm

ThefollowingfigureshowstheCalicoarchitecture:

InthefollowingtutorialwewillrunthemanualsetupofCalicoonasinglenodemachinewithDocker1.9,whichfinallybringslibnetworkoutofitsexperimentalversiontomainrelease,andCalicocanbeconfigureddirectlywithouttheneedofotherDockerexperimentalversions:

1. Gettheetcdlatestreleaseandconfigureitonthedefaultport2379:

$curl-Lhttps://github.com/coreos/etcd/releases/download/v2.2.1/etcd-

v2.2.1-linux-amd64.tar.gz-oetcd-v2.2.1-linux-amd64.tar.gz

%Total%Received%XferdAverageSpeedTimeTimeTime

Current

DloadUploadTotalSpentLeft

Speed

1006060606004450--:--:--0:00:01--:--:-

-446

1007181k1007181k00441k00:00:160:00:16--:--:-

-1387k

$tarxzvfetcd-v2.2.1-linux-amd64.tar.gz

etcd-v2.2.1-linux-amd64/

etcd-v2.2.1-linux-amd64/Documentation/

etcd-v2.2.1-linux-amd64/Documentation/04_to_2_snapshot_migration.md

etcd-v2.2.1-linux-amd64/Documentation/admin_guide.md

etcd-v2.2.1-linux-amd64/Documentation/api.md

https://github.com/projectcalico/calico-containers

contd..

etcd-v2.2.1-linux-amd64/etcd

etcd-v2.2.1-linux-amd64/etcdctl

etcd-v2.2.1-linux-amd64/README-etcdctl.md

etcd-v2.2.1-linux-amd64/README.md

$cdetcd-v2.2.1-linux-amd64

$./etcd

2016-01-0615:50:00.065733I|etcdmain:etcdVersion:2.2.1

2016-01-0615:50:00.065914I|etcdmain:GitSHA:75f8282

2016-01-0615:50:00.065961I|etcdmain:GoVersion:go1.5.1

2016-01-0615:50:00.066001I|etcdmain:GoOS/Arch:linux/amd64

Contd..

2016-01-0615:50:00.107972I|etcdserver:startingserver…[version:

2.2.1,clusterversion:2.2]

2016-01-0615:50:00.508131I|raft:ce2a822cea30bfcaisstartinganew

electionatterm5

2016-01-0615:50:00.508237I|raft:ce2a822cea30bfcabecamecandidate

atterm6

2016-01-0615:50:00.508253I|raft:ce2a822cea30bfcareceivedvote

fromce2a822cea30bfcaatterm6

2016-01-0615:50:00.508278I|raft:ce2a822cea30bfcabecameleaderat

term6

2016-01-0615:50:00.508313I|raft:raft.node:ce2a822cea30bfca

electedleaderce2a822cea30bfcaatterm6

2016-01-0615:50:00.509810I|etcdserver:published{Name:default

ClientURLs:[http://localhost:2379http://localhost:4001]}tocluster

7e27652122e8b2ae

2. OpenthenewterminalandconfiguretheDockerdaemonwiththeetcdkey-valuestorebyrunningthefollowingcommands:

$servicedockerstop

$dockerdaemon--cluster-store=etcd://0.0.0.0:2379

INFO[0000][graphdriver]usingpriorstoragedriver"aufs"

INFO[0000]APIlistenon/var/run/docker.sock


INFO[0015]Defaultbridge(docker0)isassignedwithanIPaddress

172.16.59.1/24.Daemonoption--bipcanbeusedtosetapreferredIP

address

WARN[0015]Yourkerneldoesnotsupportswapmemorylimit.


.....INFO[0034]Skippingupdateofresolv.conffilewithipv6Enabled:

falsebecausefilewastouchedbyuser



INFO[0043]Dockerdaemoncommit=a34a1d5execdriver=native-0.2

graphdriver=aufsversion=1.9.1

INFO[0043]GET/v1.21/version


INFO[0043]GET/events


3. Now,inthenewterminal,starttheCalicocontainerinthefollowingway:

$./calicoctlnode--libnetwork

NoIPprovided.UsingdetectedIP:10.22.0.1

PullingDockerimagecalico/node:v0.10.0

Caliconodeisrunningwithid:

79e75fa6d875777d31b8aead10c2712f54485c031df50667edb4d7d7cb6bb26c

PullingDockerimagecalico/node-libnetwork:v0.5.2

Calicolibnetworkdriverisrunningwithid:

bc7d65f6ab854b20b9b855abab4776056879f6edbcde9d744f218e556439997f

$dockerps



7bb7a956af37calico/node-libnetwork:v0.5.2"./start.sh"

3minutesagoUp3minutescalico-libnetwork

13a0314754d6calico/node:v0.10.0"/sbin/start_runit"

3minutesagoUp3minutescalico-node

1f13020cc3a0weaveworks/plugin:1.4.1

"/home/weave/plugin"3daysagoUp3minutes

weaveplugin

4. CreatetheCalicobridgeusingthedockernetworkcommandrecentlyintroducedintheDockerCLI:

$dockernetworkcreate–dcaliconet1

$dockernetworkls

NETWORKIDNAMEDRIVER

9b5f06307cf2docker_gwbridgebridge

1638f754fbafhosthost

02b10aaa25d7weaveweavemesh

65dc3cbcd2c0bridgebridge

f034d78cc423net1calico

5. StartthebusyboxcontainerconnectedtotheCaliconet1bridge:

$dockerrun--net=net1-itd--name=container1busybox

1731629b6897145822f73726194b1f7441b6086ee568e973d8a88b554e838366

$dockerps



1731629b6897busybox"sh"

6secondsagoUp5secondscontainer1

7bb7a956af37calico/node-libnetwork:v0.5.2"./start.sh"

6minutesagoUp6minutescalico-

libnetwork

13a0314754d6calico/node:v0.10.0"/sbin/start_runit"

6minutesagoUp6minutescalico-node

1f13020cc3a0weaveworks/plugin:1.4.1

"/home/weave/plugin"3daysagoUp6minutes

weaveplugin

$dockerattach1731

/#

/#ifconfig

cali0Linkencap:EthernetHWaddrEE:EE:EE:EE:EE:EE


inet6addr:fe80::ecee:eeff:feee:eeee/64Scope:Link





RXbytes:5774(5.6KiB)TXbytes:648(648.0B)

eth1Linkencap:EthernetHWaddr02:42:AC:11:00:02







RXbytes:4086(3.9KiB)TXbytes:648(648.0B)


inetaddr:127.0.0.1Mask:255.0.0.0







InsidethecontainerwecanseethatthecontainerisnowconnectedtotheCalicobridgeandcanconnecttotheothercontainersdeployedonthesamebridge.

SummaryInthischapter,welookedintosomeofthedeeperandmoreconceptualaspectsofDockernetworking,oneofthembeinglibnetworking,thefutureDockernetworkmodelthatisalreadygettingintoshapewiththereleaseofDocker1.9.Whileexplaininglibnetworking,wealsostudiedtheCNMmodelanditsvariousobjectsandcomponentswithitsimplementationcodesnippets.Next,welookedintodriversofCNM,theprimeonebeingtheoverlaydriver,indetail,withdeploymentaspartoftheVagrantsetup.Wealsolookedatthestand-aloneintegrationofcontainerswiththeoverlaynetworkandaswellwithDockerSwarmandDockerMachine.Inthenextsection,weexplainedabouttheCNIinterface,itsexecutableplugins,andatutorialofconfiguringDockernetworkingwiththeCNIplugin.

Inthelastsection,projectCalicoisexplainedindetail,whichprovidesascalablenetworkingsolutionbasedoutoflibnetworkandprovidesintegrationwithDocker,Kubernetes,Mesos,bare-metal,andVMsprimarily.

IndexA

AmazonEC2containerservice(AWSECS)containers,securing/SecuringcontainersinAWSECSsecuring/SecuringcontainersinAWSECS

AmazonMachineImage(AMI)/SecuringcontainersinAWSECSAppArmor/AppArmor/SELinux

used,forsecuringDockercontainers/UsingAppArmortosecureDockercontainersURL/UsingAppArmortosecureDockercontainersandDocker/AppArmorandDocker

AWSKubernetes,deploying/DeployingKubernetesonAWSMesosphere,deployingwithDCOS/DeployingMesosonAWSusingDCOS

AWSConsoleURL/DeployingKubernetesonAWS

Bbridgedriver/Driver,Bridgedriver

CCalico

libnetworkdriver/ProjectCalico’slibnetworkdriverFelix/ProjectCalico’slibnetworkdriverBIRD/ProjectCalico’slibnetworkdriverconfd/ProjectCalico’slibnetworkdrivercalicoctl/ProjectCalico’slibnetworkdriverURL/ProjectCalico’slibnetworkdriver

cgroupsabout/UnderstandingDockersecurityII–cgroupsdefining/Definingcgroupsneedfor/Whyarecgroupsrequired?manualcreation/Creatingacgroupmanuallyprocesses,attaching/AttachingprocessestocgroupsURL/Dockerandcgroupsusing,withDocker/Dockerandcgroups

CNIpluginURL/CNIpluginabout/CNIpluginversion/CNIpluginContainerID/CNIpluginnetworknamespacepath/CNIpluginnetworkconfiguration/CNIplugin,Networkconfigurationextraarguments/CNIplugininterfacename/CNIpluginresultsachieved/CNIpluginIPallocation/IPallocationIPaddressmanagementinterface/IPaddressmanagementinterface

CNMobjectsabout/CNMobjectssandbox/Sandboxendpoint/Endpointnetwork/NetworkNetworkController/Networkcontrollerattributes/CNMattributesoptionsattribute/CNMattributeslabelsattribute/CNMattributeslifecycle/CNMlifecycle

components,Kubernetesnode/Kubernetesmaster/KubernetesKubectl/KubernetesPod/Kubernetes

replicationcontroller/Kuberneteslabel/Kubernetes

CONFIG_NET_NSoptionURL/netnamespace

containernetworkinterface(CNI)/Containernetworkinterfacecontainernetworkmodel(CNM)/DesignContainerNetworkModel(CNM)/What’snewinDockernetworking?containers

andexternalnetworks,communicatingbetween/Communicationbetweencontainersandexternalnetworkscreating,withoverlaynetwork/Creatingcontainersusinganoverlaynetworkcontainernetworkinterface(CNI)/Containernetworkinterface

CoreOSonVagrantrunning,URL/Networkingwithoverlaynetworks–Flannel

Ddatacenteroperatingsystem(DCOS)

about/Mesosphereused,fordeployingMesosphereonAWS/DeployingMesosonAWSusingDCOS

DNSserverconfiguring/ConfiguringaDNSservercontainersandexternalnetworks,communicatingbetween/CommunicationbetweencontainersandexternalnetworksSSHaccess,restrictingfromonecontainertoanother/RestrictingSSHaccessfromonecontainertoanother

DockerIPstack,configuring/ConfiguringtheIPstackforDockerused,fordeployingwebapp/DeployingawebappusingDockercgroups,usingwith/DockerandcgroupsandAppArmor/AppArmorandDocker

docker0bridgeabout/Thedocker0bridge—netdefaultmode/The—netdefaultmode—net=nonemode/The—net=nonemode—net=container*$container2mode/The—net=container:$container2mode—net=hostmode/The—net=hostmodeportmapping,inDockercontainer/PortmappinginDockercontainer

Dockerbridgeconfiguring/ConfiguringtheDockerbridge

DockerCNMmodelabout/TheDockerCNMmodel

Dockercontainerslinking/LinkingDockercontainerslinks/Linksmanaging,withMarathonframework/Dockercontainerssecuring,AppArmorused/UsingAppArmortosecureDockercontainerssecuritybenchmark/Dockersecuritybenchmark

DockerHubaccountURL/Key-valuestoreinstallation

Dockermachineoverlaynetwork,usingwith/OverlaynetworkwithDockerMachineandDockerSwarm

Dockernetworkingabout/NetworkingandDockerLinuxbridges/LinuxbridgesOpenvSwitch/OpenvSwitchNAT/NAT

IPtables/IPtablesAppArmor/AppArmor/SELinuxSELnux/AppArmor/SELinuxfeatures/What’snewinDockernetworking?sandbox/Sandboxendpoint/Endpointnetwork/NetworkandKubernetesnetworking,differentiatingbetween/KubernetesnetworkinganditsdifferencestoDockernetworking

DockerOVSabout/DockerOVSVMs/DockerOVSHypervisor/DockerOVSPhysicalSwitch/DockerOVSvNIC/DockerOVSVIF(virtualinterface)/DockerOVSVirtualSwitch/DockerOVS

Dockersecuritykernelnamespaces/UnderstandingDockersecurityI–kernelnamespacescgroups/UnderstandingDockersecurityII–cgroups

DockerSwarmabout/DockerSwarmSpreadstrategy/DockerSwarmBinpackstrategy/DockerSwarmrandomstrategy/DockerSwarmsetup/DockerSwarmsetupnetworking/DockerSwarmnetworkingoverlaynetwork,usingwith/OverlaynetworkwithDockerMachineandDockerSwarm

driverabout/Drivernull/Driverbridgedriver/Driver,Bridgedriveroverlaynetworkdriver/Driver,Overlaynetworkdriverremote/Driver

dualstack/IPv6support

Eendpoint/Endpoint

Ffields,JSON

cniVersion(string)/Networkconfigurationname(string)/Networkconfigurationtype(string)/NetworkconfigurationipMasq(boolean)/Networkconfigurationipam/Networkconfigurationroutes(list)/Networkconfigurationroutes(list),dst(string)/Networkconfigurationroutes(list),gw(string)/Networkconfiguration

filesystemrestrictionsabout/Filesystemrestrictionsread-onlymountpoints/Read-onlymountpointscopy-on-write/Copy-on-write

filters,DockerSwarmconstraints/DockerSwarmaffinityfilter/DockerSwarmportfilter/DockerSwarmdependencyfilter/DockerSwarmhealthfilterfilter/DockerSwarm

Flannelabout/Networkingwithoverlaynetworks–Flannel

GGoLang

URL/UnderstandingDockersecurityI–kernelnamespaces

IIAMconsole

URL/DeployingKubernetesonAWSIPAddressManagement(IPAM)/DriverIPaddressmanagementplugin(IPAM)/IPallocationIPstack,forDocker

configuring/ConfiguringtheIPstackforDockerIPv4support/IPv4supportIPv6support/IPv6support

IPtables/IPtablesIPv4support/IPv4supportIPv6support/IPv6support

Kkernelnamespaces

about/UnderstandingDockersecurityI–kernelnamespacesPID(ProcessID)namespace/UnderstandingDockersecurityI–kernelnamespacesnetwork(net)namespace/UnderstandingDockersecurityI–kernelnamespaces,netnamespaceInterProcessCommunication(IPC)namespace/UnderstandingDockersecurityI–kernelnamespacesMount(MNT)namespace/UnderstandingDockersecurityI–kernelnamespacesUnixTimesharingSystem(UTS)namespace/UnderstandingDockersecurityI–kernelnamespacesProcessID(PID)namespace/pidnamespace

Kubernetesabout/Kubernetescomponents/Kubernetesdeploying,onAWS/DeployingKubernetesonAWSnetworking/KubernetesnetworkinganditsdifferencestoDockernetworking

KubernetesnetworkingandDockernetworking,differentiatingbetween/KubernetesnetworkinganditsdifferencestoDockernetworking

Kubernetespoddeploying/DeployingtheKubernetespod

Llibnetwork/What’snewinDockernetworking?

goal/GoalURL/Goal

libnetworkdriver,Calicoabout/ProjectCalico’slibnetworkdriver

Linuxbridges/LinuxbridgesLinuxcapabilities

about/Linuxcapabilitiescode,referencelink/LinuxcapabilitiesURL/Linuxcapabilities

MManagementConsole

URL/SecuringcontainersinAWSECSMandatoryAccessControl(MAC)/UsingAppArmortosecureDockercontainersMarathonGUI

URL/DockercontainersMesosphere

about/MesosphereDockercontainers,managing/Dockercontainerswebapp,deployingwithDocker/DeployingawebappusingDockerdeploying,onAWSwithDCOS/DeployingMesosonAWSusingDCOS

multiplecontainers,oversinglehostabout/MultiplecontainersoverasinglehostWeave,installing/WeaveyourcontainersWeave,using/Weaveyourcontainers

multiplehostOVSabout/MultiplehostOVS

Nnameserver/ConfiguringaDNSserverNAT/NATnetwork/Networknetwork(net)namespace

about/netnamespacenamespacemanagement/Basicnetworknamespacemanagementconfiguration/Networknamespaceconfiguration

NetworkController/Networkcontrollernsentercommandlineutility

URL/Creatinganewusernamespace

OOpenvSwitch/OpenvSwitchOpenvSwitch(OVS)

about/OpenvSwitchsinglehostOVS/SinglehostOVSmultiplehostOVS/MultiplehostOVS

OpenvSwitch(OVS)bridgecreating/CreatinganOVSbridge

overlaynetworkVagrant,usingwith/UsingoverlaynetworkwithVagrantdeploymentVagrantsetup/OverlaynetworkdeploymentVagrantsetupusing,withDockermachine/OverlaynetworkwithDockerMachineandDockerSwarmusing,withDockerSwarm/OverlaynetworkwithDockerMachineandDockerSwarmprerequisites/Prerequisiteskey-valuestoreinstallation/Key-valuestoreinstallationcreating/Creatinganoverlaynetworkused,forcreatingcontainers/Creatingcontainersusinganoverlaynetwork

overlaynetworkdriver/Overlaynetworkdriveroverlaynetworks/Overlaynetworksandunderlaynetworks

used,fornetworking/Networkingwithoverlaynetworks–Flannel

PPipework

about/IntroductiontoPipeworkProcessID(PID)namespace/pidnamespace

Rread-onlymountpoints

sysfsfilesystem/sysfssysfsfilesystem,URL/sysfsprocfilesystem(procfs)/procfs/dev/pts//dev/pts/sys//fs/cgroup//sys/fs/cgroup

Ssandbox/Sandboxsecuritybenchmark,Dockercontainers

about/DockersecuritybenchmarkURL/DockersecuritybenchmarkDockerdaemon,auditing/AuditDockerdaemonregularlyuser,creating/Createauserforthecontainerhostsystemdirectoriesmount,avoiding/Donotmountsensitivehostsystemdirectoriesoncontainersprivilegedcontainers,avoiding/Donotuseprivilegedcontainers

SELnux/AppArmor/SELinuxsingleOpenvSwitch(OVS)

about/SinglehostOVSbridge,creating/CreatinganOVSbridge

Swarmclustercreatingwithtwonodes/CreateaSwarmclusterwithtwonodes

Uunderlaynetworks/OverlaynetworksandunderlaynetworksUnixdomainsocket

about/Unixdomainsocketusernamespace

about/Usernamespacecreating/Creatinganewusernamespace

VVagrant

using,withoverlaynetwork/UsingoverlaynetworkwithVagrant

Wwebapp

deploying,withDocker/DeployingawebappusingDocker

learning docker networking · docker networking options are also covered in this chapter. chapter...

Documents