ipsec add-on for pcf documentation · recommendations for troubleshooting ipsec. verify that ipsec...

IPsecAdd-OnforPCF®

Documentation

Version1.8

Published:5Feb2019

©2019PivotalSoftware,Inc.AllRightsReserved.

235

11132325262830

TableofContents

TableofContentsIPsecAdd-onforPCFTroubleshootingtheIPsecAdd-onforPCFReleaseNotesInstallingtheIPsecAdd-onforPCFUpgradingtheIPsecAdd-onforPCFUninstallingtheIPsecAdd-onforPCFCheckingCertificateDatesRotatingActiveIPsecCertificatesRenewingExpiredIPsecCertificates

©CopyrightPivotalSoftwareInc,2013-2019 2 1.8

IPsecAdd-onforPCFPagelastupdated:

ThisguidedescribestheIPsecAdd-onforPCF,whichsecuresdatatransmissionsinsidePivotalCloudFoundry (PCF).TopicscoveredinthisguideincludeIPsecAdd-onforPCFinstallationandconfiguration,troubleshooting,andcertificaterotation.

YourorganizationmayrequireIPsecifyoutransmitsensitivedata.

OverviewTheIPsecAdd-onforPCFprovidessecuritytothenetworklayeroftheOSImodelwithastrongSwan implementationofIPsec.TheIPsecAdd-onprovidesastrongSwanjobinFIPSmodetoeachBOSH-deployedVM.

IPsecencryptsIPdataflowbetweenhosts,betweensecuritygateways,andbetweensecuritygatewaysandhosts.TheIPsecAdd-onforPCFsecuresnetworktrafficwithinaCloudFoundrydeploymentandprovidesinternalsystemprotectionifamaliciousactorbreachesyourfirewall.

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationabouttheIPsecAdd-onforPCF.

Element Details

Version v1.8.31

Releasedate April27,2018

CompatibleOpsManagerversion(s) v1.10.x,v1.11.x,v1.12.x,v2.0.x,andv2.1.x

CompatibleElasticRuntimeversion(s) v1.10.x,v1.11.x,andv1.12.x

CompatiblePivotalApplicationService(PAS) version(s) v2.0.xand2.1.x

IaaSsupport vSphere,GCP,AWS,Azure,andOpenstack

*AsofPCFv2.0,ElasticRuntimeisrenamedPivotalApplicationService(PAS).

IPsecImplementationDetailsTheIPsecAdd-onforPCFimplementsthefollowingcryptographicsuite:

KeyAgreement(Diffie-Hellman) IKEv2MainMode

BulkEncryption AES128GCM16

Hashing SHA2 256

Integrity/AuthenticationTag 128bitGHASHICV

DigitalSigning RSA3072/4096

PeerAuthenticationMethod Public/PrivateKey

LimitationIPsecAdd-onforPCFhasthefollowinglimitations:

Duetoaknownissue inWindowsServerOS,appshostedonPASforWindowscannotroutetrafficwhendeployedwiththeIPsecadd-onforPCF.

PivotalrecommendsconfiguringIPsectouseaself-signedcertificatetosigninstancecerts.Usingacertificatesignedbyapublicorthird-partyCAisnotrecommended.

Note:IPsecAdd-onforPCFv1.8isnolongersupported.Thesupportperiodforv1.8hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.

*


https://network.pivotal.io/products/pivotal-cfhttps://www.strongswan.org/https://github.com/Microsoft/hcsshim/issues/244

TroubleshootingtheIPsecAdd-onforPCFPagelastupdated:

ThistopicprovidesinstructionstoverifythatstrongSwan-basedIPsecworkswithyourPivotalCloudFoundry(PCF)deploymentandgeneralrecommendationsfortroubleshootingIPsec.

VerifythatIPsecWorkswithPCFToverifythatIPsecworksbetweentwohosts,youcancheckthattrafficisencryptedinthedeploymentwith tcpdump ,performthepingtest,andcheckthelogswiththestepsbelow.

1. Checktrafficencryptionandperformthepingtest.SelecttwohostsinyourdeploymentwithIPsecenabledandnotetheirIPaddresses.Thesearereferencedbelowas IP-ADDRESS-1 and IP-ADDRESS-2 .

a. SSHinto IP-ADDRESS-1 .

$sshIP-ADDRESS-1

b. Onthefirsthost,runthefollowing,andallowittocontinuerunning.

$tcpdumphostIP-ADDRESS-2

c. Fromaseparatecommandline,runthefollowing:

$sshIP-ADDRESS-2

d. Onthesecondhost,runthefollowing:

$pingIP-ADDRESS-1

e. VerifythatthepackettypeisESP.Ifthetrafficshows ESP asthepackettype,trafficissuccessfullyencrypted.Theoutputfrom tcpdump willlooksimilartothefollowing:

03:01:15.242731IPIP-ADDRESS-2>IP-ADDRESS-1:ESP(spi=0xcfdbb261,seq=0x3),length100

2. Openthe /var/log/daemon.log filetoobtainadetailedreport,includinginformationpertainingtothetypeofcertificatesyouuse,andtoverifyanestablishedconnectionexists.

3. NavigatetoyourInstallationDashboard,andclickRecentInstallLogstoviewinformationregardingyourmostrecentdeployment.Searchfor“ipsec”andthestatusoftheIPsecjob.

4. Run ipsec statusall toreturnadetailedstatusreportregardingyourconnections.Thetypicalpathforthisbinary:/var/vcap/packages/strongswan-x.x.x/sbin . x.x.x representstheversionofstrongSwanpackagedintotheIPsec.

IfyouexperiencesymptomsthatIPsecdoesnotestablishasecureconnection,returntotheInstallingtheIPsecAdd-onforPCFtopicandreviewyourinstallation.

IfyouencounterissueswithinstallingIPsec,refertotheTroubleshootingIPsecsectionofthistopic.

TroubleshootIPsec

IPsecInstallationIssues

Symptom

Unresponsiveappsorincompleteresponses,particularlyforlargepayloads


Explanation:PacketLoss

IPsecpacketencryptionincreasesthesizeofpacketpayloadsonhostVMs.Ifthesizeofthelargerpacketsexceedsthemaximumtransmissionunit(MTU)sizeofthehostVM,packetlossmayoccurwhentheVMforwardsthosepackets.

IfyourVMswerecreatedwithanAmazonPVstemcell,thedefaultMTUvalueis1500forbothhostVMsandtheapplicationcontainers.IfyourVMswerecreatedwithAmazonHVMstemcells,thedefaultMTUvalueis9001.Gardencontainersdefaultto1500MTU.

Solution

Implementa100MTUdifferencebetweenhostVMandthecontainedapplicationcontainer,usingoneofthefollowingapproaches:

DecreasetheMTUoftheapplicationcontainerstoavaluelowerthantheMTUoftheVMforthatcontainer.IntheElasticRuntimetileconfiguration,clickNetworkingandmodifyApplicationsNetworkMaximumTransmissionUnit(MTU)(inbytes)beforeyoudeploy.Decreaseitfromthedefaultvalueof1454to1354.

IncreasetheMTUoftheapplicationcontainerVMstoavaluegreaterthan1500.Pivotalrecommendsaheadroomof100.Run ifconfigNETWORK-INTERFACEmtuMTU-VALUE tomakethischange.ReplaceNETWORK-INTERFACEwiththenetworkinterfaceusedtocommunicatewithotherVMsForexample: $ifconfigNETWORK-INTERFACEmtu1600

Symptom

Unresponsiveappsorincompleteresponses,particularlyforlargepayloads

Explanation:NetworkDegradation

IPsecdataencryptionincreasesthesizeofpacketpayloads.Ifthenumberofrequestsandthesizeofyourfilesarelarge,thenetworkmaydegrade.

Solution

ScaleyourdeploymentbyallocatingmoreprocessingpowertoyourVMCPUorGPUs,which,additionally,decreasesthepacketencryptiontime.Onewaytoincreasenetworkperformanceistocompressthedatapriortoencryption.Thisapproachincreasesperformancebyreducingtheamountofdatatransferred.

IPsecRuntimeIssues

Symptom

ErrorsrelatingtoIPsec,includingsymptomsofnetworkpartition.YoumayreceiveanerrorindicatingthatIPsechasstoppedworking.

Forexample,thiserrorshowsasymptomofIPsecfailure,afailed clock_global-partition :

Failedupdatingjobclock_global-partition-abf4378108ba40fd9a43>clock_global-partition-abf4378108ba40fd9a43/0(ddb1fbfa-71b1-4114-a82c-fd75867d54fc)(canary):ActionFailedget_task:Task044424f7-c5f2-4382-5d81-57bacefbc238result:StoppingMonitoredServices:Stoppingserviceipsec:SendingstoprequesttoMonit:Requestfailed,response:Response{StatusCode:503,Status:'503ServiceUnavailable'}(00:05:22)..

Explanation:Asynchronous monit JobPriorities

WhenamonitstopcommandisissuedtotheNFSmounterjob,ithangs,preventingashutdownofthePCFcluster.

ThisisnotaproblemwiththeIPsecadd-onreleaseitself.Rather,itisaknownissuewiththeNFSmounterjobandthemonitstopscriptthatcanmanifestitselfafterIPsecisdeployedwithPCFv1.7.

Thisissueoccurswhenmonitjobprioritiesareasynchronous.Becausetheorderofjobshutdownisarbitrary,itispossiblethattheIPsecjobwillbe


stoppedfirst.Afterthishappens,thenetworkconnectivityforthatVMgoesaway,andtheNFSmounterjoblosesvisibilitytotheassociatedstorage.ThiscausestheNFSmounterjobtohang,anditblocksthemonitstopfromcompleting.SeetheMonitjobGithubdetails forfurtherinformation.

Solution1. BOSH ssh intothestuckinstancebyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier:bosh ssh VM-INDEX

ForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT -d DEPLOYMENT-NAME ssh VM-INDEX

2. Authenticateasrootandusethe svstopagent commandtokilltheBOSHAgent:

$sudosu#svstopagent

3. RunthefollowingcommandtodetectthemissingmonitjobVM.

ForOpsManagerv1.10orearlier:bosh cloudcheck

ForOpsManagerv1.11orlater: bosh2 -e ENVIRONMENT-NAME -d DEPLOYMENT-NAME cloud-check

Forexample,

#boshcloudcheckVMwithcloudID`vm-3e37133c-bc33-450e-98b1-f86d5b63502a'missing:

-Ignoreproblem-RecreateVMusinglastknownapplyspec-DeleteVMreference(DANGEROUS!)

4. Choose RecreateVMusinglastknownapplyspec .

5. Continuewithyourdeployprocedure.

SymptomAppfailstostartwiththefollowingmessage:

FAILEDServererror,statuscode:500,errorcode:10001,message:Anunknownerroroccurred.

TheCloudControllerlogshowsitisunabletocommunicatewithDiegodueto getaddrinfo failing.

Deploymentfailswithasimilarerrormessage: diego_database-partition-620982d595434269a96a/0(a643c6c0-bc43-411b-b011-58f49fb61a6f)'isnotrunningafterupdate.Reviewlogsforfailedjobs:etcd

Explanation:SplitBrain consul

Thiserrorindicatesa“splitbrain”issuewithConsul.

Solution

Confirmthisdiagnosisbycheckingthe peers.json filefrom/var/vcap/store/consul_agent/raft.Ifitisnull,thentheremaybeasplitbrain.Tofixthisproblem,followthesesteps:

Note:ThisissueaffectsdeploymentsusingCFv231orearlier,butinCFv232thereleaseusesannginxblobstoreinsteadoftheNFSblobstore.TheerrordoesnotexistforPCFdeploymentsusingCFreleasesgreaterthanCFv231.TheerroralsodoesnotapplytoPCFdeploymentsthatuseWebDAVastheirCloudControllerblobstore.


https://github.com/cloudfoundry/cf-release/blob/v231/jobs/nfs_mounter/monit

1. Run monit stop onallConsulservers:

2. Run rm -rf /var/vcap/store/consul_agent/ onallConsulservers.

3. Run monit start consul_agent onallConsulserversoneatatime.

4. Restarttheconsul_agentprocessontheCloudControllerVM.Youmayneedtorestartconsul_agentonotherVMs,aswell.

Symptom

YouseethatcommunicationisnotencryptedbetweentwoVMs.

Explanation:ErrorinNetworkConfiguration

TheIPsecBOSHjobisnotrunningoneitherVM.ThisproblemcouldhappenifbothIPsecjobscrash,bothIPsecjobsfailtostart,orthesubnetconfigurationisincorrect.ThereisamomentarygapbetweenthetimewhenaninstanceiscreatedandwhenBOSHsetsupIPsec.Duringthistime,datacanbesentunencrypted.Thislengthoftimedependsontheinstancetype,IAAS,andotherfactors.Forexample,onat2.microonAWS,thetimefromnetworkingstarttoIPsecconnectionwasmeasuredat95.45seconds.

Solution

SetupanetworkingrestrictiononhostVMstoonlyallowIPsecprotocolandblockthenormalTCP/UDPtraffic.Forexample,inAWS,configureanetworksecuritygroupwiththeminimalnetworkingsettingasshownbelowandblockallotherTCPandUDPports.

AdditionalAWSConfiguration

Type Protocol PortRange Source

CustomProtocol AH(51) All 10.0.0.0/16

CustomProtocol ESP(50) All 10.0.0.0/16

CustomUDPRule UDP 500 10.0.0.0/16

Symptom

Youseeunencryptedappmessagesinthelogs.

Explanation: etcd SplitBrain

Solution1. CheckforsplitbrainetcdbyconnectingwithBOSH ssh intoeachetcdnode:

$curllocalhost:4001/v2/members

2. Checkifthemembersareconsistentonallofetcd.Ifanodehasonlyitselfasamember,ithasformeditsownclusteranddeveloped"splitbrain."Tofixthisissue,SSHintothesplitbrainVMandrunthefollowingcommands:

a. $sudosu-

b. #monitstopetcd

Note:Whenconfiguringanetworksecuritygroup,IPsecaddsanadditionallayertotheoriginalcommunicationprotocol.Ifacertainconnectionistargetingaportnumber,forexampleport8080withTCP,itactuallyusesIPprotocol50/51instead.Duetothisdetail,traffictargetedatablockedportmaybeabletogothrough.


c. #rm-r/var/vcap/store/etcd

d. #monitstartetcd

3. Checkthelogstoconfirmthenoderejoinedtheexistingcluster.

Symptom

IPsecdeploymentfailswith Errorfillingintemplate'pre-start.erb'

Error100:Unabletorenderinstancegroupsfordeployment.Errorsare:-Unabletorenderjobsforinstancegroup'consul_server-partition-f9c4b18fd83cf3114d7f'.Errorsare:-Unabletorendertemplatesforjob'ipsec'.Errorsare:-Errorfillingintemplate'pre-start.erb'(line12:undefinedmethod`each_with_index'for#)-Unabletorenderjobsforinstancegroup'nats-partition-f9c4b18fd83cf3114d7f'.Errorsare:-Unabletorendertemplatesforjob'ipsec'.Errorsare:-Errorfillingintemplate'pre-start.erb'(line12:undefinedmethod`each_with_index'for#)

Explanation:TypographicalorsyntaxerrorindeploymentdescritorYAMLsyntax

Solution

CheckthedeploymentdescriptorYAMLsyntaxfortheCAcertificatesentry:

releases:-{name:ipsec,version:1.0.0}

addons:-name:ipsec-addonjobs:-name:ipsecrelease:ipsecproperties:ipsec:ipsec_subnets:-10.0.1.1/20no_ipsec_subnets:-10.0.1.10/32#boshdirectorinstance_certificate:|-----BEGINCERTIFICATE-----MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw...-----ENDCERTIFICATE-----instance_private_key:|-----BEGINEXAMPLERSAPRIVATEKEY-----MIIEogIBAAKCAQEAtAkBjrzr5x9g0aWgyDEmLd7m9u/ZzpK7UScfANLaN7JiNz3c...-----ENDEXAMPLERSAPRIVATEKEY-----ca_certificates:-|-----BEGINCERTIFICATE-----MIIEUDCCArigAwIBAgIJAJVLBeJ9Wm3TMA0GCSqGSIb3DQEBCwUAMB0xGzAZBgNVBAMMElBDRiBJUHNlYyBBZGRPbiBDQTAeFw0xNjA4MTUxNzQwNDVaFw0xOTA4MTUx...-----ENDCERTIFICATE-----

Intheexampleabove,thevaluesthatappearafterthe ca_certificates :keyarecontainedwithinalistandarenotjustasinglecertificate.Thisentrymustbefollowedbyalinestartingwith - ,andendingwith | .ThelinesfollowingthiscontainthePEMencodedcertificate(s).

Theerrormessageshownaboveindicatingaproblemwiththe each_with_index methodprovidesahintthatthe -| YAMLsyntaxsequenceismissing.UsethissyntaxeveninsituationswherethereisonlyoneCAcertificate,forexamplealistofoneentry.

Symptom


Completesystemoutagewithnowarning.

Explanation:IPsecCertificatesMightHaveExpired

ExpiredIPseccertificatescancauseasuddensystemoutage.Forexample,theself-signedcertificatesgeneratedbythescriptprovidedintheinstallationinstructionshavealifetimeof365days.IPseccertificatesexpireifyoudonotrotatethemwithintheirlifetime.

Solution

RenewexpiredIPseccertificates.ToavoidfuturedowntimeduetoexpiredIPseccertificates,setacalendarremindertorotatethecertificatesbeforetheyexpire.

Forhowtorenewcertificates,seeRenewingExpiredIPsecCertificates.Forhowtorotatethem,seeRotatingIPsecCertificates.

Symptom

BOSHshowsaVMinafailingstate.OnthefailingVM, monitsummary

shows ipsec withastatusof Doesnotexist .

Explanation:IPsecstopped/crashedandMonitCannotAutomaticallyBringitBackUp.

IPsecisrequiredtobethefirstprocesstostartandthelastprocesstostop.Asaresult,thestartandstopscriptsarelocatedin pre-start and post-stop ,whichisaconcepttoBOSH,butnottoMonit.MonitisnotabletobringIPsecbackupautomaticallybecauseitdoesnotknowwhat pre-start is.Afteryourunthe pre-start manually,thenMonitisabletodetectIPsecashealthy.

Solution1. FromthefailingVM,runthefollowingcommandasroot: /var/vcap/jobs/ipsec/bin/pre-start

2. Run monit summary .Iftherestartissuccessful,Monitshows ipsec withastatusof Running .


ReleaseNotesPagelastupdated:

ThistopiccontainsreleasenotesfortheIPsecAdd-onforPCF.

v1.8.31ReleaseDate:April27,2018

Featuresincludedinthisrelease:

OptionstoconfiguresyslogmessagesthatwarnaboutupcomingIPseccertificatesexpiry.

Fixedissuesinthisrelease:

WhenthehostVMhasmorethanonenetworkinterfaceavailable,the leftsubnet isalwaysconfiguredwiththeinternalIPaddress.Previously,theleftsubnet wasconfiguredwiththefirstavailableinterface,whichmayormaynotbetheinternaladdress.

v1.8.14ReleaseDate:January24,2018


Anewmanifestpropertynamed dpdaction hasbeenadded.ThispropertycontrolstheIPsecresponseupondetectinga“dead”networkpeer.Thedefaultvalueis restart .

Knownissuesinthisrelease:

WhenusingIKEv1,Pivotalrecommendsthatyousetthemanifestproperty dpdaction to none .

v1.8.12ReleaseDate:January19,2018


AnarbitrarylengthcertificatechainisnowsupportedforbothLinuxandWindows.

AwarningmessageisgeneratedintheIPsecstdoutlogfilewhenthe optional flagissettotrue.

Anewmanifestpropertyhasbeenadded, optional_warn_interval ,tocontrolthemessageintervalfortheoptional-is-truewarning.

Thedefault log_level isnowsetto-1forLinux.

Thegolangdependencyhasbeenupdatedtov1.9.2.

AnerrorisreportedifthehostIPaddressisinneithertheIPsecnortheno-IPsecsubnet.

If,uponshutdown,theIPsecjobisnotthelastmonitjobrunning,awarningislogged.

Anewmanifestpropertyhasbeenadded, ike_version .Acceptedvaluesare ikev1 or ike (forIKEv2).

AdditionalcertificateverificationhasbeenaddedsothatanerrorisreportedifthesuppliedinstancecertificateandCAcertificatearenotrelated.


PivotaldoesnotrecommendusingIKEv1becauseofsecurityandperformancelimitations.

OnlyuseIKEv1fordeploymentsthatrequireit:mixedenvironmentscontainingbothLinuxandWindowsVMs.

v1.8.3ReleaseDate:October11,2017



UpdatesstrongSwanto5.6.0

UpdatesOpenSSLto1.0.2l

UpdatesOpenSSLFIPSto2.0.16

Loglevelisnowconfigurable.

Keyexchangeisnowconfigurable.

InstancecertificateisvalidatedwiththeCAcertificateonstart.

Fixedissuesinthisrelease:

Stopscripttimeoutisconfigurable.


IKEv1onWindows:WindowsusesIKEv1forKeyexchange.IKEv2doesnotsupportmultiplerootcertificates,andthereforedoesnotsupportcertificaterotation.AnissuehasbeenfiledwithMicrosoft.

SpuriousConfigurationWarning:AspartoftheupgradetoStrongSwanv5.4.0,thisversionoftheIPsecadd-onmayemitasequenceofspuriousconfigurationwarningmessages.Themessagesaresimilartothefollowing:

!!Yourstrongswan.confcontainsmanualpluginloadoptionsforcharon.!!Thisisrecommendedforexpertsonly,see!!http://wiki.strongswan.org/projects/strongswan/wiki/PluginLoad

Thesemessagesarebothexpectedandharmless.Asacautiontoendusers,theStrongSwansoftwarenowemitsawarningmessagewhenitdetectsthattheinstallationincludesamanuallyconfiguredsetofplugins.Asamatterofsecurityhygienebestpractices,theIPsecadd-onhasalwaysusedamanual(explicit)configurationandloadsarestrictedsetofStrongSwanplugins.Anyunusedpluginsarenotloaded.ThenewestversionofStrongSwannowissuesthiswarningmessagewhenitdetectsthatsituation.TheactuallistofpluginsinusehasbeendeterminedtobeappropriateforuseofStrongSwaninthePCFenvironment.Thiswarningisexpectedandshouldbeignored.

MTUSizing:Use1354onOpenStack.KeepthedefaultonAWSandvSphere.


InstallingtheIPsecAdd-onforPCFPagelastupdated:

ThistopicdescribeshowtoprepareyournetworkforIPsec,createanIPsecruntimeconfig,andaddIPsectoyourdeployment.

PrerequisitesTocompletetheIPsecinstallation,verifythatyouhavesatisfiedthefollowingprerequisitesbeforeyoubegin:

GoogleCloudPlatform(GCP),vSphere,Azure,AmazonWebServices(AWS),orOpenStackasyourIaaS

PivotalCloudFoundry(PCF)operatoradministrationrights

BOSHdeployedthroughOpsManagerv1.8orlater

SettheMTUforyourIaaSinthePivotalApplicationService(PAS)orElasticRuntimetile,underNetworking.PivotalrecommendsMTUvaluesof1354onGCP,1438onAzure,andthedefaultvaluesonAWSandvSphere.ForOpenStack,followtherecommendationsofyourNeutron/ML2 pluginprovider,orempiricallytestthecorrectMTUforyourenvironment.

BestPracticesIPsecmayaffectthefunctionalityofotherservicetiles.Asaresult,PivotalrecommendsdeployingPAS(orElasticRuntime)andeachservicetiletodifferentisolatedsubnets.Alternatively,youcanminimallydeployallservicetilestoasingleisolatedsubnet,apartfromthePAS(orElasticRuntime)subnet.SomeservicetilesdonotsupportIPsecandmustbeplacedinanon-IPsecsubnet.

ForIPseconLinuxVMs,PivotalrecommendsanyUbuntustemcellsforvSphere,OpenStack,andHVMstemcellsforAWS.ThesestemcellsareavailableonPivotalNetwork .IfyouusePVstemcellsobtainedfrombosh.io ,seethePacketLosssectionoftheTroubleshootingtheIPsecAdd-onforPCFtopictoadjustMTUvalues.

ForIPseconWindowsVMs,PivotalrecommendstheWindows2012R2stemcellsforAWS,GCP,orAzureavailableonPivotalNetwork .

Step1:ConfigureNetworkSecurityPerformthestepsintheappropriatesectionbelowtoconfigureyourIaaSnetworksecurity.

GoogleCloudPlatformToconfigureyourGoogleCloudPlatform(GCP)environmentforIPsec,performthefollowingsteps:

1. NavigatetotheNetworkingsectionoftheGCPConsole.

2. ClickFirewallrules.

3. ClickCreateFirewallRule.

4. ForName,enter ipsec .

5. ForNetwork,selectthenetworkwhereOpsManagerisdeployed.Forexample,opsmgr.

6. ForSourcefilter,selectAllowfromanysource(0.0.0.0/0).

7. ForAllowedprotocolsandports,enter udp:500; ah; esp .

8. ClickCreate.

9. AdjusttheMTUvalueto 1354 byperformingtheprocedureinthePacketLosssectionoftheTroubleshootingtheIPsecAdd-onforPCFtopic.

vSphere


https://wiki.openstack.org/wiki/Neutron/ML2https://network.pivotal.io/products/stemcellshttps://bosh.iohttps://network.pivotal.io/products/stemcells-windows-server

Confirmthatyournetworkallowstheprotocolslistedinthetablebelow.

ProtocolName ProtocolNumber Port(s)

AH 51 Any

ESP 50 Any

UDP 17 500

Azure1. Confirmthatyournetworkallowstheprotocolslistedinthetablebelow.

ProtocolName ProtocolNumber Port(s)

AH 51 Any

ESP 50 Any

UDP 17 500

2. AdjusttheMTUvalueto 1438 .Forinstructions,seeExplanation:PacketLoss.

AWSToconfigureyourAWSenvironmentforIPsec,performthefollowingsteps:

1. NavigatetoEC2Dashboard>SecurityGroups.

2. SelecttheSecurityGroupwiththedescriptionPCFVMsSecurityGroupandclickEdit.

3. CreatethefollowingInboundRules.

Type ProtocolName ProtocolNumber PortRange Source

CustomProtocol AH 51 All 10.0.0.0/16

CustomProtocol ESP 50 All 10.0.0.0/16

CustomUDPRule UDP 17 500 10.0.0.0/16

OpenStack

ToconfigureyourMirantisOpenStackenvironmentforIPsec,performthefollowingsteps:

1. NavigatetoProject/Access&Security.

2. SelectthesecuritygroupandclickManageRules.

3. CreatethefollowingIngressandEgressRules.AdjustthesourceCIDRasneededforyourenvironment.

ProtocolName ProtocolNumber PortRange Source

ESP 50 Any 0.0.0.0/0

AH 51 Any 0.0.0.0/0

UDP 17 500 0.0.0.0/0

Note:ThedefaultPCFVMsSecurityGroupistypicallyspecifiedwithasubnetof 10.0.0.0/16 .IfyourPCFsubnetisdeployedtoadifferentCIDRblock,adjustthesourceasneeded.

Note:ThefollowingnetworkconfigurationisoptimizedforMirantisOpenStack,butotherOpenStackdistributionshaveasimilarworkflow.


Step2:CreatetheIPsecManifestToaddIPsectoVMsinyourdeployment,youmustcreatearuntimeconfigfilenamed ipsec-addon.yml thatconfiguresIPsecadd-onpropertiesforLinuxVMs,WindowsVMs,orboth.Performthefollowingsteps:

1. CreateanIPsecruntimeconfigfile ipsec-addon.yml ,withthecodebelowasatemplate.

releases:\- name: ipsec version: 1.X.Xaddons:

1. Addpropertiestothe ipsec-addon.yml fileasdescribedbelowforLinuxVMsandWindowsVMs.

AddLinuxVMSupporttoYourRuntimeConfigPerformthefollowingstepstoaddIPsectoLinuxVMsinyourdeployment:

1. AddthefollowingYAMLunder addons: toyour ipsec-addon.yml file:

releases:- name: ipsec version: 1.X.Xaddons:- name: ipsec-addon jobs: - name: ipsec release: ipsec include: stemcell: - os: ubuntu-trusty properties: ipsec: optional: false ipsec_subnets: - 10.0.1.1/20 no_ipsec_subnets: - 10.0.1.10/32 # bosh director - 10.0.1.4/32 # ops manager instance_certificate: | -----BEGIN CERTIFICATE----- MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw ... -----END CERTIFICATE----- instance_private_key: | -----BEGIN EXAMPLE RSA PRIVATE KEY----- EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA ... -----END EXAMPLE RSA PRIVATE KEY----- ca_certificates: - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0 ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATAAYDVQQDEwl0ZXN0NBgkqhkiG9w0BAQsFADAUMRIwE ... -----END CERTIFICATE----- prestart_timeout: 30 esp_proposals: aes128gcm16! ike_proposals: aes128-sha256-modp2048! log_level: 1 ike_version: ike optional_warn_interval: 1 force_udp_encapsulation: false instance_certificate_info_period: 30 instance_certificate_warn_period: 14 instance_certificate_error_period: 7 instance_certificate_critical_period: 3

Note:EnablingIPsecforWindowsaddsIPsecsecuritytoWindowsVMsthatuserscancreateafterinstallingthePASforWindows2012R2 tile.


https://docs.pivotal.io/pivotalcf/windows2012r2/index.html

2. Replacethevalueslistedinthetemplateasfollows:

releases: - version :SpecifytheversionnumberofyourIPsecdownloadfromPivotalNetwork. optional :ThisvaluemakesIPsecenforcementoptional.ToaddIPsectoanexistingPAS(orElasticRuntime)deployment,setthisflagtotrue .AfterIPsechasbeensuccessfullyinstalled,setthisflagbackto false andredeploy.

ipsec_subnets :Listthesubnetsthatyouwanttobeencrypted.Youcanincludetheentiredeploymentoraportionofthenetwork.Encryptanynetworkthathandlesbusiness-sensitivedata. no_ipsec_subnets :ListtheIPaddressofyourBOSHDirectorandOpsManagerVM,alongwithanyotherIPaddressesinyourPCF

deploymentthatyouwanttocommunicatewithwithoutencryption.PivotalrecommendsthatyoulistthesubnetsthatareusedforPCFmanagedservices.SubnetsforPCFmanagedservicesthatdonotsupportIPsec(suchasanPivotalOpsManager)mustbelistedunderno_ipsec_subnets .

instance_certificate :CopyinthesignedcertificatethatwillbeusedbyallyourinstanceVMs.YoumustuseoneoftheCAsintheca_certificates propertytosignthiscertificate.Pivotalrecommendsthatyouuseaself-signedcertificate.Formoreinformation,seeGeneratea

Self-SignedCertificate.

instance_private_key :Copyintheprivatekeythatcorrespondstothe instance_certificate above.Thiskeymustnotuseapassphrase.

ca_certificates :CopyinCAcertificatesfortheinstanceVMtotrustduringthevalidationprocess.Inmostcases,youonlyneedtheCAcertificateusedtosigntheinstancecertificate.DuringCAcredentialrotation,youneedtwoCAcertificates.

IPsecv1.8.12andlatersupportstheCAcertificatechain.Concatenatethecontentsoftherootandtheintermediatecertificatesasoneofthelistitemsinca_certificates,withtherootCAatthetop:

ca_certificates: - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

prestart_timeout :Youcanmodifythe30-seconddefaultprestarttimeoutvalue.ThisvaluelimitsthenumberofsecondsallowedforIPsectostartbeforefailingtheattempt.

log_level :YoucanspecifytheIKEdaemonnumericalloglevel,rangingfrom -1 to 4 .Formoreinformation,seeLoggerConfiguration inthestrongSwandocumentation.

optional_warn_interval :Theintervalinhoursofwarningwhen optional propertyissettotrue.Itprintsthewarningmessage{Date} - IPsec is set to "Optional" inthefile /var/vcap/sys/log/ipsec/ipsec.stdout.log forLinux.

force_udp_encapsulation :AvailableonLinux-onlydeployments.Ifsetto true itforcesUDPencapsulationforESPpackets.

WARNING:CommunicationbetweenexistingcomponentsfailsifyoutrytoaddIPsectoanexistingdeploymentwithoutsettingoptional to true .

Note:IfyouhaveanexternalloadbalancersuchasF5,addittothe no_ipsec_subnets property.Ifyouwanttoincludeitinthe ipsec_subnet ,youmustconfigureitmanually.

WARNING:IPsthatarenotin ipsec_subnets or no_ipsec_subnets havenodefaultbehaviorandcannotcommunicatewithotherinternalVMs.YoumustspecifyinternalIPsin ipsec_subnets or no_ipsec_subnets .

WARNING:InGCP,ifyouusethedefaultrouterforDNSinsteadoftheGooglepublicDNSat 8.8.8.8 ,youmustaddtheIPaddressofthedefaultrouterinyoursubnetto no_ipsec_subnets .Forexample, 10.0.0.1/32 .

Note:TherootandtheintermediatecertificatescannothavethesamesubjectName.Thisisalsocalledthecommonname,andissetwithCN= .Therootmustbethefirstcertificateofthechain.


https://wiki.strongswan.org/projects/strongswan/wiki/LoggerConfiguration

instance_certificate_info_period :Iftheinstancecertificateexpiresduringthesetperiodindays,theIPsecreleasewritesan [INFO] messageinthelogs.

instance_certificate_warn_period :Iftheinstancecertificateexpiresduringthesetperiodindays,theIPsecreleasewritesa [WARN] messageinthelogs.

instance_certificate_error_period :Iftheinstancecertificateexpiresduringthesetperiodindays,theIPsecreleasewritesan [ERROR]messageinthelogs.

instance_certificate_critical_period :Iftheinstancecertificateexpiresduringthesetperiodindays,theIPsecreleasewritesa [CRITICAL]messageinthelogs.

AddWindowsVMSupporttoYourRuntimeConfigToaddIPsectoWindowsServer2012R2VMsinyourdeployment,followthesesteps:

1. Modifythe ipsec-addon.yml createdduringtheprevioussectiontoaddthepropertiesindicatedinboldbelowunderthe ipsec key.

- name: ipsec-addon ... properties: ipsec: . . . ike_version: ikev1 dpdaction: none

1. AddthefollowingYAMLunder addons: toyour ipsec-addon.yml file.Additunderthe ipsec-addon sectionforLinux,ifyouincludedoneabove.

- name: ipsec-windows-addon jobs: - name: ipsec-win release: ipsec include: stemcell: - os: windows2012R2 properties: ipsec: optional: false ipsec_subnets: - 10.0.1.1/20 no_ipsec_subnets: - 10.0.1.10/32 # bosh director - 10.0.1.4/32 # ops manager instance_certificate: | -----BEGIN CERTIFICATE----- MIIEMDCCAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw ... -----END CERTIFICATE----- instance_private_key: | -----BEGIN EXAMPLE RSA PRIVATE KEY----- EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA ... -----END EXAMPLE RSA PRIVATE KEY----- ca_certificates: - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0 ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIBATAAYDVQQDEwl0ZXN0NBgkqhkiG9w0BAQsFADAUMRIwE ... -----END CERTIFICATE----- quick_mode_proposals: - encryption: AESGCM128 hash: AESGMAC128 main_mode_proposals:

WARNING:Settingthispropertyto true inmixeddeploymentscausesthedeploymenttofail.IfyoudonothaveaLinux-onlydeployment,youmustset force_udp_encapsulation to false .


- encryption: AES128 hash: SHA256 keyexchange: DH14

1.Replacethevalueslistedinthetemplateasfollows:* ipsec_subnets :Copyandpastethevaluefromipsec_subnetsforLinux.* no_ipsec_subnets :Copyandpastethevaluefromno_ipsec_subnetsforLinux.* instance_certificate :Copyandpastethevaluefrom

instance_certificateforLinux.* instance_private_key :Copyandpastethevaluefrominstance_private_keyforLinux.* ca_certificates :Copyandpastethevaluefromca_certificatesforLinux.* optional :CopyandpastethevaluefromoptionalforLinux.

Optional:CustomLinux/WindowsMixedDeploymentProposalsAdefaultproposalsetisalreadyselectedforthe ipsec-addon.yml .Ifyouwanttousedifferentproposals,modifythe ipsec-addon.yml usingthefollowingtable:

1. Selecttheencryptiontypefromthefirstrow.

2. Copythepropertiesfromthatrowinto ipsec-addon.yml accordingly.Seethe ipsec-addon.yml fileexampleabove.

EncryptionTypeLinux(ipsec-addon) Windows(ipsec-win-addon)

ike_proposals esp_proposals main_mode_proposals quick_mode_proposals

128BitEncryption aes128-sha256-modp2048! aes128gcm16!- encryption: AES128 hash: SHA256 keyexchange: DH14

- encryption: AESGCM128 hash: AESGMAC128

256BitEncryption aes256-sha256-modp2048! aes256gcm16!- encryption: AES256 hash: SHA256 keyexchange: DH14


ike_proposals :YoucanmodifytheIKE(MainMode)encryptionandintegrityalgorithms,andtheDiffie-Hellmangroup.Thedefault,aes128-sha256-modp2048! ,is128bitAES-CBCforencryption,SHA2_256_128HMACforintegrity,andGroup14forDiffie-Hellman.

esp_proposals :YoucanmodifytheESP(QuickMode)encryptionandintegrityalgorithms.Thedefault, aes128gcm16! ,is128bitAES-GCMwith128bitICVforbothencryptionandintegrity.

main_mode_proposals :ThisisanarrayofMainModealgorithmsforencryption,integrity,andkeyexchange.Thisvaluemustmatchthelistspecifiedinike_proposalsforLinux.SeethetableforproposalsetsforbothLinuxandWindows.ThedefaultentrythatmatchestheLinuxdefaultis:

- encryption: AES128 hash: SHA256 keyexchange: DH14

quick_mode_proposals :ThisisanarrayofQuickModealgorithmsforencryptionandintegrity.Thisvaluemustmatchthelistspecifiedinesp_proposalsforLinux.SeethetableforproposalsetsforbothLinuxandWindows.ThedefaultentrythatmatchestheLinuxdefaultis:


Step3:DownloadandDeploytheIPsecAdd-onTodownloadtheIPsecbinary,addyourIPsecruntimeconfigtoyourBOSHmanifest,anddeploytheIPsecadd-on,followtheprocedurebelow.

AssumptionaboutOpsManagerVersionsTheprocedurebelowassumesthefollowingaboutyourOpsManager,BOSHCLI,andruntimeconfig.

WARNING:CommunicationbetweenexistingcomponentsfailsifyoutrytoaddIPsectoanexistingdeploymentwithoutsetting optional to true .


OpsManagerVersion BOSHCLIVersion RuntimeConfig Moreinformation1.10andearlier CLIv1 single,default,runtimeconfigfile BOSHCLIv1

1.11andlater CLIv2+ runtimeconfiginmultiple,namedfiles,sothat ipsec canbemanagedseparatelyBOSHCLIv2

Configs-bosh

Procedure1. DownloadtheIPsecadd-onsoftwarebinaryfromthePivotalNetwork toyourlocalmachine.

2. CopythesoftwarebinarytoyourOpsManagerinstance.

$scp-iPATH-TO-PRIVATE-KEYipsec-release.tar.gzubuntu@YOUR-OPS-MANAGER-VM-IP:

3. CopytheIPsecruntimeconfigfiletoyourOpsManagerinstance.

$scp-iPATH-TO-PRIVATE-KEYipsec-addon.ymlubuntu@YOUR-OPS-MANAGER-VM-IP:

4. SSHintoOpsManager.

$ssh-iPATH-TO-PRIVATE-KEYubuntu@YOUR-OPS-MANAGER-VM-IP

5. OntheOpsManagerVM,navigatetothesoftwarebinarylocationinyourworkingdirectory.

$cdPATH-TO-BINARY

6. LogintotheBOSHDirector.

ForOpsManagerv1.10orearlier:

i. OntheOpsManagerVM,targettheinternalIPaddressofyourBOSHDirector.Whenprompted,enteryourBOSHDirectorcredentials.ToretrieveyourBOSHDirectorcredentials,navigatetoOpsManager,clicktheCredentialstab,andclickLinktoCredentialnexttoDirectorCredentials.Forexample:

$bosh--ca-cert/var/tempest/workspaces/default/root_ca_certificatetargetYOUR-BOSH-DIRECTOR-INTERNAL-IPTargetsetto'p-bosh'Yourusername:directorEnterpassword:******************Loggedinas'director'

ForOpsManagerv1.11orlater:

i. OntheOpsManagerVM,createanaliasintheBOSHCLIforyourOpsManagerDirectorIPaddress.Forexample:

$bosh2alias-envmy-env-e10.0.0.3

ii. LogintotheBOSHDirector,specifyingthenewlycreatedalias.Forexample:

$bosh2-emy-envlog-in

7. Uploadyourrelease,specifyingthepathtothetarballedIPsecbinary,byrunningoneofthefollowingcommands:


$boshuploadreleasePATH-TO-BINARY/BINARY-NAME.tar


$bosh2-emy-envupload-releasePATH-TO-BINARY/BINARY-NAME.tar

BreakingChange:IfyouareusingPCFv1.11orlater,youmustusenamedruntimeconfigs.Ifyouhavenotalreadysplityourruntimeconfigintomultiplenamedfiles,dosobeforeupgradingtheIPsecAdd-onforPCF.Forgeneralinformationaboutnamedruntimeconfigfiles,seeConfigs .


https://bosh.io/docs/sysadmin-commands.htmlhttps://bosh.io/docs/cli-v2.htmlhttps://bosh.io/docs/configs.htmlhttps://bosh.io/docs/configs.htmlhttps://network.pivotal.io/products/p-ipsec-addon

8. Listthereleasesbyrunningoneofthefollowingcommands,andconfirmthattheIPsecbinaryfileappears:


$boshreleases


$bosh2-emy-envreleases

9. Downloadyourcurrentruntimeconfigandsaveas bosh-manifest.yml byrunningoneofthefollowingcommands:


$boshruntime-config>bosh-manifest.yml


$bosh2-emy-envruntime-config>bosh-manifest.yml

10. ForOpsManagerv1.10orearlier:AppendthecontentsofyourIPsecmanifest ipsec-addon.yml to bosh-manifest.yml .

11. UpdateyourruntimeconfigurationtoincludetheIPsecadd-on.


$boshupdateruntime-configPATH/bosh-manifest.yml


$bosh2-emy-envupdate-runtime-config--name=ipsecPATH/bosh-manifest.yml

12. VerifyyourruntimeconfigurationchangesmatchwhatyouspecifiedintheIPsecmanifestfile.


$boshruntime-config


$bosh2-emy-envruntime-config--name=ipsec

Forexample:

$bosh2-emy-envruntime-config--name=ipsecActingasuser'admin'on'micro'

releases:-{name:ipsec,version:1.0.0}

addons:name:ipsec-addonjobs:-name:ipsecrelease:ipsec...-name:ipsec-win#ifusingWindowsrelease:ipsec...

13. IfyouhavealreadydeployedPAS(orElasticRuntime)orareaddingIPsectoanexistingdeployment:

a. Setthe optional flagto true .b. NavigatetoyourInstallationDashboardinOpsManager.c. ClickApplyChangesd. Waitfortheinstallationtocomplete.e. Setthe optional flagto false .f. Updatetheruntimeconfig.



$boshupdateruntime-configPATH/bosh-manifest.yml


$bosh2-emy-envupdate-runtime-config--name=ipsecPATH/bosh-manifest.yml

g. NavigatetoyourInstallationDashboard.h. ClickApplyChanges.

14. IfthePAS(orElasticRuntime)tileisnotyetinstalled:

a. NavigatetoyourInstallationDashboardinOpsManager.b. ClickApplyChangesc. DeployPAS(orElasticRuntime)byfollowingtheinstallationinstructionsforyourIaaS.Formoreinformation,seeInstallingPivotalCloud

Foundry .

15. The bosh-manifest.yml and ipsec-addon.yml filescontainsensitiveinformation.Whenthedeploymentprocessiscompleted,besuretoremoveanyunneededcopiesofthesefilesfromthelocalworkstation.Pivotalrecommendsthatanyarchivalcopiesofmanifestfilestoberetainedshouldbeappropriatelysecuredviaencryptionand/orlogicalaccesscontrols.

Step4:VerifyYourIPsecInstallationAfterinstallingIPsecanddeployingPAS(orElasticRuntime),performthefollowingstepstoverifyyourIPsecinstallation:

1. ListthejobVMsinyourdeploymentbyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier:bosh vms

ForOpsManagerv1.11orlater:bosh2 -e BOSH-ENVIRONMENT vms

2. OpenanSSHconnectionintotheVM,usingthejobnameandindexofanyVMfoundabove,byrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier:bosh ssh JOB-NAME/INDEX

ForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT -d DEPLOYMENT-NAME ssh JOB-NAME/INDEX

3. Run sudosu- toentertherootenvironmentwithrootprivileges.

4. Run monitsummary toconfirmthatyour ipsec jobislistedasa bosh job.

TheMonitdaemon5.2.5uptime:18h32m...Process'ipsec'runningSystem'system_localhost'running

5. Run PATH-TO-IPSEC/ipsecstatusall toconfirmthatIPsecisrunning.IfIPsecisnotrunning,thiscommandproducesnooutput.

Note:TheexactVMdoesnotmatter,becauseinstallingtheIPsecadd-onloadsIPseconallVMsdeployedbyOpsManager.


http://docs.pivotal.io/pivotalcf/installing/index.html

$/var/vcap/packages/strongswan-5.3.5/sbin/ipsecstatusallStatusofIKEcharondaemon(strongSwan5.3.5,Linux3.19.0-56-generic,x86_64):uptime:18hours,sinceMar1623:58:502016malloc:sbrk2314240,mmap0,used1182400,free1131840workerthreads:11of16idle,5/0/0/0working,jobqueue:0/0/0/0,scheduled:206loadedplugins:charonaessha1sha2randomnoncex509revocationconstraintspubkeypkcs1pkcs7pkcs8pkcs12pemgmpxcbccmachmacattrkernel-netlinksocket-defaultstrokeListeningIPaddresses:10.10.5.66Connections:ipsec-10.10.4.0/24:%any...%anyIKEv1/2ipsec-10.10.4.0/24:local:[CN=test-cert-1-ca-1]usespublickeyauthenticationipsec-10.10.4.0/24:cert:"CN=test-cert-1-ca-1"ipsec-10.10.4.0/24:remote:usespublickeyauthenticationipsec-10.10.9.0/24:child:10.10.5.66/32===10.10.9.0/24TRANSPORTno-ipsec-10.10.4.1/32:%any...%anyIKEv1/2no-ipsec-10.10.4.1/32:local:usespublickeyauthenticationno-ipsec-10.10.4.1/32:remote:usespublickeyauthenticationno-ipsec-10.10.4.1/32:child:dynamic===10.10.4.1/32PASSShuntedConnections:no-ipsec-10.10.4.1/32:dynamic===10.10.4.1/32PASSno-ipsec-10.10.5.1/32:dynamic===10.10.5.1/32PASSno-ipsec-10.10.6.1/32:dynamic===10.10.6.1/32PASSRoutedConnections:ipsec-10.10.9.0/24{6}:ROUTED,TRANSPORT,reqid6ipsec-10.10.9.0/24{6}:10.10.5.66/32===10.10.9.0/24ipsec-10.10.8.0/24{5}:ROUTED,TRANSPORT,reqid5ipsec-10.10.4.0/24{1}:10.10.5.66/32===10.10.4.0/24SecurityAssociations(45up,0connecting):ipsec-10.10.4.0/24[459]:ESTABLISHED13secondsago,10.10.5.66[CN=test-cert-1-ca-1]...10.10.4.38[CN=test-cert-1-ca-1]ipsec-10.10.4.0/24{1527}:10.10.5.66/32===10.10.4.38/32...

6. IfyouinstalledIPsecforWindows,followthesesteps:

a. FromanyWindowsVM,openWindowsFirewallwithAdvancedSecurity.b. ClickConnectionSecurityRules.c. Confirmthatyouseerulesforeach ipsec and no-ipsec subnetthatyoulistedinyourmanifest.

GenerateaSelf-SignedCertificatewithOpenSSLFollowthesestepstogenerateaself-signedcertificateforyourIPsecmanifest.

1. Download the openssl-create-ipsec-certs.sh bashscript.

2. Navigatetothedirectorywhereyoudownloadedthescript:

$cd~/workspace

3. Changethepermissionsofthescript:

$chmodu+xopenssl-create-ipsec-certs.sh

4. Runthescript:

$./openssl-create-ipsec-certs.sh

5. Thisgeneratesfourfilesinanew certs directorywherethescriptisrun:

pcf-ipsec-ca-cert.pem—thisvaluecanbeusedastheCACertinthe ca_certificates manifestfield.pcf-ipsec-ca-key.pem—thekeyusedtosignthegeneratedCACert.pcf-ipsec-peer-key.pem—thisvaluecanbeusedastheinstanceprivatekeyinthe instance_private_key manifestfield.pcf-ipsec-peer-cert.pem—thisvaluecanbeusedastheinstancecertificateinthe instance_certificate manifestfield.

6. Becausethiscertificateexpiresin365days,setacalendarremindertorotatethecertificatewithintheyear.Forinstructionsonchangingcertificates,seeRotatingIPsecCertificates.


https://docs.pivotal.io/addon-ipsec/1-8/scripts/openssl-create-ipsec-certs.sh

UpgradingtheIPsecAdd-onforPCFPagelastupdated:

ThistopicdescribeshowtoupgradetheIPsecAdd-onforPCF.

AssumptionaboutOpsManagerVersionsThistopicassumesthefollowingaboutyourOpsManager,BOSHCLI,andruntimeconfig.

OpsManagerVersion BOSHCLIVersion RuntimeConfig Moreinformation

1.10andearlier CLIv1 single,default,runtimeconfigfile BOSHCLIv1

1.11andlater CLIv2+ runtimeconfiginmultiple,namedfiles,sothat ipsec canbemanagedseparatelyBOSHCLIv2

Configs-bosh

UpgradeIPsecAdd-onToupgradetheIPsecadd-ontoalaterversion,dothefollowing:

1. DownloadtheIPsecadd-onsoftwarebinaryfromthePivotalNetwork toyourlocalmachine.

2. TocopythesoftwarebinarytoyourOpsManagerVM,runthefollowingcommand:

scp-iPATH-TO-PRIVATE-KEYipsec-VERSION.tar.gzubuntu@YOUR-OPS-MANAGER-VM-IP:

Forexample:

$cp-i~/.ssh/my-key.pem~/Downloads/[email protected]:

3. SSHintotheOpsManagerVM.Forhowtodothis,seeSSHintoOpsManager .

4. Retrievethelatestruntimeconfigbyrunningoneofthefollowingcommands:


boshruntime-config>PATH-TO-SAVE-THE-RUNTIME-CONFIG

Forexample:

boshruntime-config>/tmp/ipsec.yml


bosh2-eBOSH-ENVIRONMENTruntime-config--nameipsec>PATH-TO-SAVE-THE-RUNTIME-CONFIG

Forexample:

bosh2-emy-envruntime-config--nameipsec>/tmp/ipsec.yml

5. UploadthelatestIPsecrelease:


BreakingChange:IfyouareusingPCFv1.11orlater,youmustusenamedruntimeconfigs.Ifyouhavenotalreadysplityourruntimeconfigintomultiplenamedfiles,dosobeforeupgradingtheIPsecAdd-onforPCF.Forgeneralinformationaboutnamedruntimeconfigfiles,seeConfigs .


https://bosh.io/docs/sysadmin-commands.htmlhttps://bosh.io/docs/cli-v2.htmlhttps://bosh.io/docs/configs.htmlhttps://bosh.io/docs/configs.htmlhttps://network.pivotal.io/products/p-ipsec-addonhttps://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#ssh

boshuploadreleasePATH-TO-NEW-IPSEC-RELEASE

Forexample:

boshuploadrelease~/ipsec-1.8.14.tgz


bosh2-eBOSH-ENVIRONMENTupload-releasePATH-TO-NEW-IPSEC-RELEASE

Forexample:

bosh2-emy-envupload-release~/ipsec-1.8.14.tgz

6. Edittheipsecruntimeconfigtosetthenewreleaseversion.Forexample,edittheversionin /tmp/ipsec.yml asfollows:

releases:- {name: ipsec, version: 1.8.14}

7. Updatetheruntimeconfig:


boshupdateruntime-configPATH-TO-SAVE-THE-RUNTIME-CONFIG

Forexample:

boshupdateruntime-config/tmp/ipsec.yml


bosh2-eBOSH-ENVIRONMENTupdate-runtime-config--name=ipsecPATH-TO-SAVE-THE-RUNTIME-CONFIG

Forexample:

bosh2-emy-envupdate-runtime-config--name=ipsec/tmp/ipsec.yml

8. NavigatetoyourInstallationDashboardinOpsManager.

9. ClickApplyChanges.


UninstallingtheIPsecAdd-onforPCFPagelastupdated:

ThistopicdescribeshowtouninstallIPsecfromyourdeployment.

UninstalltheIPsecAdd-On1. Retrievethelatestruntimeconfigbyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier: bosh runtime-config > PATH-TO-SAVE-THE-RUNTIME-CONFIGForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT runtime-config > PATH-TO-SAVE-THE-RUNTIME-CONFIG

2. Setthe optional flagto true underIPsecproperties.

3. Updatetheruntimeconfigbyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier: bosh update runtime-config PATH/YOUR-RUNTIME-CONFIG.ymlForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT update-runtime-config --name=ipsec PATH-TO-SAVE-THE-RUNTIME-CONFIG



6. Waitfortheinstallationtocomplete.

7. RemoveIPsecfromtheruntimeconfig.


ForOpsManagerv1.10orearlier: bosh update runtime-config PATH/YOUR-RUNTIME-CONFIG.ymlForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT update-runtime-config --name=ipsec PATH-TO-SAVE-THE-RUNTIME-CONFIG




CheckingCertificateDatesPagelastupdated:

ThistopicdescribeshowtochecktheexpirationdatesofIPseccertificates.

ThefollowingproceduredescribeshowtodownloadtheruntimeconfigurationfileandextractthetwoIPseccertificatesintotemporaryfiles.Then,thefilesareinputtotheOpenSSLtool.TheOpenSSLtooldecodesthecertificatesanddisplaystheexpirationdates.

CheckCertificateDatesFollowthestepsbelowtodeterminetheexpirationdatesofyourIPseccertificates.

1. LogintoBOSHDirector.

2. RunoneofthefollowingcommandstodownloadyourruntimeconfigurationYAMLfile:

ForOpsManagerv1.10orearlier: bosh runtime-config > PATH-TO-SAVE-THE-RUNTIME-CONFIGForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT runtime-config --name=ipsec > PATH-TO-SAVE-THE-RUNTIME-CONFIG

Forexample,

boshruntime-config>/tmp/my-runtime-config.yml

3. DisplaytheruntimeconfigurationYAMLfilesothatyoucancopyfromit.Forexample,

$cat/tmp/my-runtime-config.yml

4. IdentifythesectionofthefilethatcontainsIPsecproperties,andlocatethecertificates:

addons:-include:stemcell:-os:ubuntu-trustyjobs:-name:ipsecrelease:ipsecname:ipsecproperties:ipsec:ca_certificates:-|-----BEGINCERTIFICATE-----MIIE/TCCAuWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAOMQwwCgYDVQQDEwNjYTEwHhcNMTYwNTI2MjI1MDMzWhcNMjYwNTI2MjI1MDQyWjAOMQwwCgYDVQQDEwNjYTEw...Axu2pbEoT1PrMd3HlAZ3AH8ZrMR3ScJKCW3wQFRX/Plj-----ENDCERTIFICATE-----instance_certificate:|-----BEGINCERTIFICATE-----MIIEGTCCAgGgAwIBAgIQDlqK1V54BEknnblVPXu5lzANBgkqhkiG9w0BAQsFADAOMQwwCgYDVQQDEwNjYTEwHhcNMTYwNTI2MjI1MTAzWhcNMTgwNTI2MjI1MTAzWjAQMQ4wDAYDVQQDEwVjZXJ0MTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB...4Q6P/cDn9QvW2QbbWkApP2uuMk04jWJV7p79CfX4pipPqiSofjFyFqsjjvir-----ENDCERTIFICATE-----

5. Copytheca_certificateintoatextfile.Retaintheheaderandfooter,butdeletetheleadingwhitespacebeforethe -----BEGINCERTIFICATE----- and -----ENDCERTIFICATE----- lines.Forexample,

-----BEGINCERTIFICATE-----MIIE/TCCAuWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAOMQwwCgYDVQQDEwNjYTEwHhcNMTYwNTI2MjI1MDMzWhcNMjYwNTI2MjI1MDQyWjAOMQwwCgYDVQQDEwNjYTEw...Axu2pbEoT1PrMd3HlAZ3AH8ZrMR3ScJKCW3wQFRX/Plj-----ENDCERTIFICATE-----


6. SavethefilewiththePEMextension,forexample, my-ipsec-ca-cert.pem .

7. Runthefollowingcommand:opensslx509-text-informpem-in/PATH/FILENAME.pem|grep"NotAfter"

Where /PATH/FILENAME.pem isthepathtoandfilenameofthefileyousavedinthestepabove.Forexample,

$opensslx509-text-informpem-in/tmp/my-ipsec-ca-cert.pem|grep"NotAfter"NotAfter:May2622:50:422026GMT

IfthePEMfileiscorrectlyformatted,theoutputshowsalinewiththe NotAfter date.IfthePEMfileisnotcorrectlyformatted,Theoutputshows unabletoloadcertificate .

8. Repeatsteps5–7fortheinstance_certificate.

9. Reviewthe NotAfter dateandplantoreplacethecertificatesaccordingly.Keepinmindtheleadtimetoobtainnewcertificatesandthetimetoperformadeploymenttoapplythem.Forinformation,seeRotatingActiveIPsecCertificates.

10. Forsecurityhygiene,deletethreetemporaryfilesthatyoucreated:thedownloadedcopyofthe runtime-config.yml whichcontainstheprivatekeyandthetwoPEMfilesthatcontainthecertificates.


RotatingActiveIPsecCertificatesPagelastupdated:

ThistopicdescribestheprocessPivotalrecommendstoincreasedeploymentsecuritybyrotatingcertificatesintheIPsecmanifest.

WhyYouNeedtoRotateCredentialsThesearecommonreasonsforrotatingcredentials:

Yourorganizationalsecuritypolicymayspecifyhowoftenyoushouldapplythesechanges.

Yourcertificatesaregoingtoexpire.Tofindtheexpirationdatesonyourcertificates,seeCheckingCertificateDates.

AbouttheProceduresTherearetwoproceduresforcertificaterotationdescribedinthistopic:

Procedure1describesrotatingthefollowingcertificatesspecifiedinyourIPsecmanifest:

TheinstancecertificateandinstanceprivatekeyThisprocedurerequiresupdatingBOSH.Itdoesnotincluderotatingthecertificateauthority(CA)certificate.

Procedure2describesrotatingyourCAcertificateinadditiontoyourinstancecertificateandinstanceprivatekey.ThisprocedurerequiresupdatingBOSHthreetimes.

Procedure1:RotatetheInstanceCertificateandInstancePrivateKeyFollowthestepsbelowtorotatetheinstancecertificateandinstanceprivatekey.

1. GenerateanewcertificateanduseyourexistingIPsecCAcertificatetosignthenewcertificate.

2. Updatetheinstancecertificateandtheprivatekeyfieldsinyour ipsec-addon.yml filewithnewvaluesfromthepreviousstep.


ForOpsManagerv1.10orearlier: bosh update runtime-config PATH-TO-SAVE-THE-RUNTIME-CONFIGForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT update-runtime-config --name=ipsec PATH-TO-SAVE-THE-RUNTIME-CONFIG

4. NavigatetoyourOpsManagerinterfaceinabrowser,andclickApplyChanges.

Procedure2:RotatetheCACertificate,theInstanceCertificate,andInstancePrivateKeyFollowthesestepstorotatetheCAcertificate,instancecertificate,andinstanceprivatekey.

1. GenerateanewCAcertificate.

2. AppendthenewlygeneratedCAcertificateundertheexistingcertificateasanewyamllistelementinyour ipsec-addon.yml .Forexample:

ca_certificates: - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Note:Therollingdeploysduringtheseproceduresresultinminimaldeploymentdowntime.

Note:Thisstepresultsinafewminutesofappdowntime.


- | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- . . .

Forv1.8.12andabove:IPsecsupportsCAcertificatechain.

Concatenatethecontentsoftherootandtheintermediatecertificatesasoneofthelistitemsinca_certificates(therootCAisatthetop).

ca_certificates: - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----




5. GenerateanewcertificateanduseyournewCAcertificatetosignthenewcertificate.

6. Updatetheinstancecertificateandtheprivatekeyfieldsintheyour ipsec-addon.yml filewithnewvaluesfromabove.

7. Repeatstep3toupdatetheruntimeconfig.


9. DeletetheolderCAcertificateinthe ipsec-addon.yml file.

10. Repeatstep3toupdatetheruntimeconfig.


Note:TherootandtheintermediatecertificatescannothavethesamesubjectName,(alsocalledthecommonnameandsetwith CN= ).Also,therootcertificatemustbethefirstcertificateofthechain.

Note:Thisstepresultsinafewminutesofappdowntime.


RenewingExpiredIPsecCertificatesPagelastupdated:

ThistopicdescribesthebasicprocessthatdeployersmayusetorenewanyalreadyexpiredcertificatescontainedintheIPsecmanifest.

AboutCertificateExpirationTheIPsecAdd-onreliesuponX.509certificatestosecurethecommunicationsbetweencommunicatingpeers.

Likeallcertificates,theIPseccertificateshaveafinitelifetimeandeventuallyexpire.Thecertificatesgeneratedbytheprocedureprovidedintheinstallationinstructions,GenerateaSelf-SignedCertificatehaveadefaultlifetimeofoneyear.Regardlessoftheirspecificlifetime,allcertificatesmusteventuallyberotated,andsoitisimportantfortheoperationsteamtoplanaccordinglyandremembertorotatetheIPseccertificatesbeforetheyactuallyexpire.

RenewExpiredIPsecCertificatesTorenewexpiredIPseccertificates,dothefollowing:

1. Retrievethelatestruntimeconfigbyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier: bosh runtime-config > PATH-TO-SAVE-THE-RUNTIME-CONFIGForOpsManagerv1.11orlater: bosh2 -e BOSH-ENVIRONMENT runtime-config > PATH-TO-SAVE-THE-RUNTIME-CONFIG

2. Generateanewsetofcertificates.Fordevelopmentortestenvironments,youcanuseself-signedcertificates.Forinformationaboutself-signedcertificates,seeGenerateaSelf-SignedCertificate.

3. Intheruntime config.yml filesavedfromstep1,updatethe optional fieldto true andupdatethecertificatefieldswithnewcertificates.Formoreinformationaboutthesefields,seethefielddescriptionsunderCreatetheIPsecManifest.

properties: ipsec: optional: true instance_certificate: | -----BEGIN CERTIFICATE----- EXAMPLEAhigAwIBAgIRAIvrBY2TttU/LeRhO+V1t0YwDQYJKoZIhvcNAQELBQAw ... -----END CERTIFICATE----- instance_private_key: | -----BEGIN EXAMPLE RSA PRIVATE KEY----- EXAMPLExRSAxPRIVATExKEYxDATAxEXAMPLExRSAxPRIVATExKEYxDATA ... -----END EXAMPLE RSA PRIVATE KEY----- ca_certificates: - | -----BEGIN CERTIFICATE----- ExampleAvGgAwIBAgIBATANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDEwl0ZXN0 ... -----END CERTIFICATE-----





7. Removethe optional:true setinstep3.

8. Repeatsteps4to6.

IMPORTANT:RotatingthecertificateswhiletheyarestillvalidensuresthemaximumavailabilityoftheCloudFoundryplatformandavoidsanyunscheduledinterruptioninservice.



Table of ContentsIPsec Add-on for PCFOverviewProduct SnapshotIPsec Implementation DetailsLimitation

Troubleshooting the IPsec Add-on for PCFVerify that IPsec Works with PCFTroubleshoot IPsecIPsec Installation IssuesSymptomExplanation: Packet LossSolutionSymptomExplanation: Network DegradationSolution

IPsec Runtime IssuesSymptomExplanation: Asynchronous monit Job PrioritiesSolutionSymptomExplanation: Split Brain consulSolutionSymptomExplanation: Error in Network ConfigurationSolutionSymptomExplanation: etcd Split BrainSolutionSymptomExplanation: Typographical or syntax error in deployment descritor YAML syntaxSolutionSymptomExplanation: IPsec Certificates Might Have ExpiredSolutionSymptomExplanation: IPsec stopped/crashed and Monit Cannot Automatically Bring it Back Up.Solution

Release Notesv1.8.31v1.8.14v1.8.12v1.8.3

Installing the IPsec Add-on for PCFPrerequisitesBest PracticesStep 1: Configure Network SecurityGoogle Cloud PlatformvSphereAzureAWSOpenStack

Step 2: Create the IPsec ManifestAdd Linux VM Support to Your Runtime ConfigAdd Windows VM Support to Your Runtime ConfigOptional: Custom Linux/Windows Mixed Deployment Proposals

Step 3: Download and Deploy the IPsec Add-onAssumption about Ops Manager VersionsProcedure

Step 4: Verify Your IPsec InstallationGenerate a Self-Signed Certificate with OpenSSL

Upgrading the IPsec Add-on for PCFAssumption about Ops Manager VersionsUpgrade IPsec Add-on

Uninstalling the IPsec Add-on for PCFUninstall the IPsec Add-On

Checking Certificate DatesCheck Certificate Dates

Rotating Active IPsec CertificatesWhy You Need to Rotate CredentialsAbout the ProceduresProcedure 1: Rotate the Instance Certificate and Instance Private KeyProcedure 2: Rotate the CA Certificate, the Instance Certificate, and Instance Private Key

Renewing Expired IPsec CertificatesAbout Certificate ExpirationRenew Expired IPsec Certificates

ipsec add-on for pcf documentation · recommendations for troubleshooting ipsec. verify that ipsec...

Documents