recent bgp innovations for operational challenges · 2019. 10. 1. · recent bgp innovations for...

RecentBGPInnovationsforOperationalChallenges

[email protected]

NTTLtd.

AusNOG2019,Melbourne 1

Background

• There’sbeenincreasedparticipationbyoperatorsintheIETFrecentlytostandardizesolutionstooperationalchallengeswithBGP– IDR (Inter-DomainRouting)WorkingGroup– GROW (GlobalRoutingOperations)WorkingGroup

• SeveralRFCshavebeenpublished,andseveralI-Dsareinthestandardizationprocess– OperatorsandimplementersareworkingonsolutionstogetherintheWGs

• ThispresentationprovidesanoverviewofsomeoftherecentinnovationsinBGP

• It’snevertoolatetoparticipate,jointheIDR andGROWmailinglists!– https://www.ietf.org/wg/


Agenda


Performance

RFC8212:ApplysecureEBGPpolicydefaults

RFC8326:Reducepacketlossthroughcooperation

BCP214:Reducepacketlossthroughcorrectprocedures

Safety

ManagementRFC8092:Signallargecommunitieswith32-bitASNs

RFC8203:SendfreeformmessagewithBGPshutdown

Coordination

BGP

RFC8212“DefaultExternalBGP(EBGP)RoutePropagationBehaviorwithoutPolicies”


Whatdoesthisconfigurationdo?


router bgp 64499!neighbor 192.0.2.1 remote-as 64555neighbor 192.0.2.1 description Upstream 1!neighbor 192.0.2.5 remote-as 65444neighbor 192.0.2.5 description Upstream 2

!

LateralAS-AS-ASLeak


AS64555 AS64500

AS64499

RFC8212inaNutshell


OpponentsArgued

• “Wecan’tchangedefaults”• “Itcan’tbedone”• ”Itwillbreakeverythingweloveandknow”• Customersdon’treadreleasenotes– Anddon’ttestwhetherthesoftwareboots• Anddeploynewsoftwareabsolutelyeverywhereatonce

– Anddon’tfollowcommunitymailinglists» Anddon’ttalktoeachother

• …..


Post-RFC8212implication(hypothetical)


route-map implicit-deny-all deny 1!router bgp 64499!neighbor 192.0.2.1 remote-as 64555neighbor 192.0.2.1 description Upstream 1neighbor 192.0.2.1 route-map implicit-deny-all inneighbor 192.0.2.1 route-map implicit-deny-all out!neighbor 192.0.2.5 remote-as 65444neighbor 192.0.2.5 description Upstream 2neighbor 192.0.2.5 route-map implicit-deny-all inneighbor 192.0.2.5 route-map implicit-deny-all out

AdvantagesofRFC8212

• Protectsyournetwork– Consistencyacrossplatforms&vendors– Explicitconfiguration(‘grep’ suddenlyisusefulagain)– Handoverbetweenpersonneliseasieraswedon’thavetoguess

• ProtectstheDefault-FreeZone(EBGPisasharedresource)– Becauserelyingonotherswillbreaksomeday


WhatThisMeans• BGPspeakersthatannounceroutesand/oracceptroutes,withoutexplicitlybeing

configuredtodoso,arenolongercompliantwiththecoreBGPspecification• FixedVendors

– BIRD– OpenBGPD– NokiaSROS

• Currentlistofvendorsthatneedtodosomework– CiscoIOS– CiscoIOSXE– CiscoNX-OS– AristaEOS– JuniperJunos OS– BrocadeIronware

• We’rekeepingtrackhere,withworkaroundshttps://github.com/bgp/RFC8212


UsageGuidelines

• StarttoimplementaroutingpolicywithsecureEBGPdefaultsnow– It’stherightthingtodoandnowisagoodtimetostart

• KeepaneyeoutforwhenyourBGPimplementationschangetheirdefaultbehavior– Checkreleasenotesanddocumentation

• Followingthesestepswillensureyouarepreparedinadvance


Agenda


Performance




Safety



Coordination

BGP

NeededRFC1997StyleCommunities,butLarger

• ASN:Community form• Weknewwe’drunoutof16-bitASNseventually

andcameupwith32-bitASNs• RIRsstartedallocating32-bitASNsbyrequestin

2007,nodistinctionbetween16-bitand32-bitASNsnow

• However,youcan’tfita32-bitvalueintoa16-bitfield

• Can’tusenative32-bitASNswithRFC1997communities

• NeededanInternetroutingcommunitiessolutionfor32-bitASNsforalmost10years

• ParitysoeveryonecanusetheirgloballyuniqueASN


RFC8092“BGPLargeCommunitiesAttribute”

• IdeaprogressedrapidlyfrominceptioninMarch2016

• FirstI-DinSeptember2016toRFCpublicationonFebruary16,2017injustsevenmonths

• Finalstandard,plusanumberofimplementationandtoolsdevelopedaswell

• Networkoperatorscantestanddeploythenewtechnologynow

AusNOG2019,MelbourneCakeandphotocourtesyoftheNTTCommunicationsNOC.

15

GettingStartedWithLargeCommunities

• 2018istheyearoflargeBGPcommunities– Preparation,testing,traininganddeploymentcantakeweeks,monthsorevenoverayear– Starttheworknow,soyouarereadywhencustomerswanttouselargecommunities

• Lotsofresourcesareavailabletohelpnetworkoperatorslearnaboutlargecommunitiesathttp://largebgpcommunties.net/– BGPspeakerimplementations– Analysisandecosystemtools– Presentations(http://largebgpcommunities.net/talks/)– Documentationforeachimplementation– Configurationexamples(http://largebgpcommunities.net/examples/)– RFC8195 providesexamplesandinspirationfornetworkoperatorstouselarge

communitiesAusNOG2019,Melbourne 16

Agenda


Performance




Safety



Coordination

BGP

CommunicationcanbeaChallenge


https://labs.ripe.net/Members/cteusche/bgp-meets-cat19AusNOG2019,Melbourne

RFC8203“BGPAdministrativeShutdownCommunication”

• Coordinationproblem:youshutdownyourBGPsessionandyourpeersdon’tknowwhy

• Solution:addafreeformmessageembeddedintheBGPNOTIFICATIONmessagewhenthesessionisshutdown


AS64496 AS64511X ProbablyDidn’tReadMaintenanceNotice?Maintenance ??

RFC8203“BGPAdministrativeShutdownCommunication”

• Messagecanbeupto128byteslong• UTF-8issupportedtoo:💩🦄😍😡👭😺👬


☺💡

AS64496 AS64511XMaintenance

NOTIFICATIONCease"[TICKET-1-1438367390]softwareupgrade;backin2hours”

UsageGuidelines

Sender• Send“AdministrativeShutdown”

messageformaintenancethatisgoingtotakesomeperiodoftime

• Send“AdministrativeReset”messageformaintenancethatisforashorttime,forexampletoresetapeerortorebootarouter

• Includeaticketorreferencenumberandmakethemessageasinformativeaspossible

Receiver• Logmessagestologgingsystems• Referenceticketnumberinemailor

othernotificationsformoredetails


OpenBGPD ExampleSender:[job@kiera ~]$ bgpctl neighbor 165.254.255.24 down "[TICKET-1-1438367390] we are upgrading to openbsd 6.1, be back in 30 minutes”[job@kiera ~]$

Receiver:

Jan 8 19:28:54 shutdown bgpd[50719]: neighbor 165.254.255.26: received notification: Cease, administratively down

Jan 8 19:28:54 shutdown bgpd[50719]: neighbor 165.254.255.26: received shutdown reason: "[TICKET-1-1438367390] we are upgrading to openbsd 6.1, be back in 30 minutes"


ImplementationStatus


Implementation Software Status

cz.nic BIRD Unknown

Cisco IOSXR Unknown

ExaBGP ExaBGP ✔ Done!

FreeRangeRouting frr ✔ Done!

OSRG GoBGP ✔ Done!

Juniper Junos OS Unknown

Nokia SROS Unknown

OpenBSD OpenBGPD ✔ Done!

OSRG GoBGP ✔ Done!

pmacct.net pmacct ✔ Done!

tcpdump.org tcpdump ✔ Done!

Wireshark Dissector ✔ Done!

Agenda


Performance




Safety



Coordination

BGP

TwoTypesofMaintenance

VoluntaryShutdown(YOU)• Youtakeactionbeforemaintenanceto

reroutetrafficandminimizetheimpact

Ø YouuseBGPshutdowncommunicationØ YouusegracefulBGPsession

shutdown

InvoluntaryShutdown(otherfolks)• Maintenanceonlowerlayernetwork

breaksend-to-endpath,butlinkstaysup

• BGPsessionsonlygodownafterholdtimerexpires

• Couldblackholetrafficduringthistimeuntiltrafficisrerouted

Ø YournetworkproviderusesBGPculling


Whendoesblackholing happenwithvanillashutdown?

• Lackofanalternativerouteonsomerouters• Transientroutinginconsistency• Aroutereflectormayonlypropagateitsbestpath

• ThebackupASBRmaynotadvertisethebackuppathbecausethenominalpathispreferred

Admittedly,theabovescenariosusuallyareshortperiodsofblackholing,butwhyacceptthatiftheycaneasilybeprevented?

ASBR

RR

ASBR

Peer

1)Shutdown(cease)

2)withdraw3)Newpath

Steadyannounce

Steadyannounce


GracefulShutdowntriggers“pathhunting”

ASBR

RR

ASBR

Peer

1)Signal“lowerLOCAL_PREF”4)shutdownsession(cease)

2)ANNOUNCEwithLP=03)ReceiveNewpathfromRR

Steadyannounce

• InitiatedbytheoperatorontherouterbeforemaintenancebysendingtheGRACEFUL_SHUTDOWNwell-knowncommunity(65535:0asperIANA)

• ReceivingEBGPpeersetsLOCAL_PREFERENCEto0andselectspathstoroutetrafficawayfromtheinitiator,(similartosettingoverloadinanISIS)

• WhenBGPsessiongoesdown,minimizesimpacttotrafficbecausealternatepathshavealreadybeeninstalled


UsageGuidelines• Tosupportreceivinggracefulshutdown,updateyourroutingpolicyto– MatchtheGRACEFUL_SHUTDOWNwell-knowncommunity(65535:0)– SettheLOCAL_PREFattributetoalowvalue,like0

• Tosendgracefulshutdown,updateyourroutingpolicyto– SendtheGRACEFUL_SHUTDOWNwell-knowncommunity(65535:0)beforeyoustartmaintenance

– Wheningresstrafficfromthepeerhasstopped,startmaintenanceanduseBGPshutdowncommunication

– RemovetheGRACEFUL_SHUTDOWNwell-knowncommunitywhenyouaredone


ConfigurationExample– SimpletoImplement


route-policy AS64497-ebgp-inbound

if community matches-any (65535:0) thenset local-preference 0

endif

end-policy

!

router bgp 64496

neighbor 2001:db8:1:2::1remote-as 64497

address-family ipv6 unicast

send-community-ebgp

route-policy AS64497-ebgp-inbound in

ip community-list standard gshut 65535:0!route-map ebgp-in permit 10

match community gshutset local-preference 0

Arista/Brocade/IOS/Quagga/FRRIOSXR

community "gshut" members "65535:0"policy-statement "ebgp-in"

entry 10 from

community "gshut"exit action accept

local-preference 0exit

exit exit

Nokia

GRACEFUL_SHUTDOWNsignals:

“Helloeveryone,ifyouconsiderthispathyour‘bestpath’,pleasestartconsideringthispaththe’worstpath’

andifyoufindanythingbetterinstallthatintoyourFIB.Thispathwilldisappearwithinafewminutes.”


Operatorsknowntohonorthegraceful_shutdown well-knowncommunity:

• NTT(AS2914)• GTT(AS3257)• Github (AS36459)

• Nordunet (AS2603)• Coloclue (AS8283)• Amsio (AS8315)• BIT(AS12859)• Telia(AS3301/1299)• Tele2(AS1257)

• SVT(AS201641)• Netnod(AS8674)• Bahnhof(AS8473)• DGCSystems(AS21195)• ComHem(AS39651)• … you?J

BCP214“MitigatingNegativeImpactofMaintenancethrough

BGPSessionCulling”

• Performanceproblem:maintenanceonlowerlayernetworkbreakspath,butlinkstaysup

• Solution:networkproviderappliesLayer4ACLstoblockBGPcontrolplanetrafficwhilelinksareup

• Routerscontinuetoforwardtrafficuntilholdtimerexpires,noblackholing


BGPisupUntilHoldTimerExpires


AS64496 AS64511Layer2Network

XMaintenance

LinkupTrafficisDropped

LinkupTrafficisDropped

BCP214“MitigatingNegativeImpactofMaintenancethrough

BGPSessionCulling”

• LowerlayernetworkproviderappliesLayer4ACLstoblockBGPcontrolplanetrafficbeforemaintenancestarts

• Dataplanecontinuestoforward• WhenBGPholdtimerexpires,BGPchoosesanewpath• Thenlowerlayernetworkstartsmaintenance,andremovesACLswhenmaintenanceis

complete




AS64496 AS64511Layer2Network

XMaintenance

ACLBlocksBGPTrafficisForwarded

ACLBlocksBGPTrafficisForwarded

“InvoluntaryTeardown”UsageGuidelines

• ACLsareonlyappliedtoTCP/179ondirectlyconnectedIPaddresses–Multihop BGPcontrolplanetrafficispermitted– Dataplanetrafficispermitted

• ACLsareappliedtoIPv4andIPv6IPaddresses• Maintenanceisstartedwhendataplanetraffichasstoppedordroppedsignificantly

• ACLsareremovedaftermaintenance


AvailabilityOverview

• Configchanges:– GracefulShutdown– BGPSessionCulling

• Codechanges:– LargeBGPCommunities– ShutdownCommunication– EBGPSecureDefaults


CalltoAction


YourBGPSoftwareSuppliers

• AskthemtosupportthefollowingRFCsnow,evenifit’salreadylistedontheirroadmap– RFC8092 BGPLargeCommunities– RFC8203 BGPAdministrativeShutdownCommunication– RFC8212 DefaultEBGPRoutePropagationBehaviorwithoutPolicies

• WhenyouwriteaRequestForProposals(RFP),makesurethesethreeitemsareonthechecklist

• Votewithyourwallet


YourPeers,TransitProviders&IXPs:

• Askyourtransitproviders&peerstosupport– RFC8326 GracefulBGPsessionshutdown– BCP214 VoluntaryShutdownBCP

• AskIXPstoapplyBGPculling(orequivalent)duringmaintenance– BCP214 (InvoluntaryShutdownBCP)- MitigatingNegativeImpactofMaintenancethroughBGPSessionCulling

• WhenyouwriteaRequestForProposals(RFP),makesurethesethreeitemsareonthechecklist.PUTTHISINRFPs!

• Votewithyourwallet


YourNetwork

• Updateyourroutingpolicy– AssumeSecureEBGPdefaults– GRACEFUL_SHUTDOWNwell-knowncommunity(65535:0)– Largecommunities– Documentandpublishit

• Addcoordinationandperformanceimprovementstoyourmaintenanceprocedures– ShutdowncommunicationandBGPgracefulshutdown– FollowBGPsessioncullingBCP


MovieCredits(contributorstoRFC8092,8195,8203,8212)

Acee LindemAdamChappellAdamDavenportAdamRoachAdamSimpsonAlexanderAzimovAlvaroRetanaArjenZonneveldArnoldNipperBarryO'DonovanBenMaddisonBertrandDuvivierBillFennerBradDreisbachBrianDicksonBrunoDecraeneChristophDietzelChristopherMorrowDaleWorleyDavidFarmer

DavidFreedmanDonaldSmithDuncanLockwoodEduardoAscenco ReisGaurab RajUpadhayaGeoffHustonGert DoeringGregHankinsGregSkinnerGrzegorz JanoszkaGuntervandeVeldeIanDickinsonIgnas BagdonasJakob HeitzJamesBensleyJanBaggenJaredMauchJayBorkenhagenJeffHaasJeffTantsura

JeffreyHaasJobSnijdersJoeProvoJoelM.HalpernJohnHeasleyJohnScudderJonathanStewartJulianSeifertJussi PeltolaKayRechthienKeyur PatelKristianLarssonLindaDunbarLouBergerMachChenMarcoDavidsMarcoMarzettiMarkSchoutenMarkusHauschildMartijn Schmidt

MartinMillnertMikaelAbrahamssonNabeel CockerNickHilliardNielsBakkerPaulHoogstederPeterHesslerPetervanDijkPierCarloChiodiRandyBushRemco vanMookRichardHartmannRichardSteenbergenRichardTurkbergenRobShakirRobertRaszukRuediger VolkRussWhiteSaku YttiSanderSteffann

ShaneAmanteShawnMorrisShyam SethuramSriram KotikalapudiStefanPlugStewartBryantSusanHaresTeun VinkTheodoreBaschakThomasKingTomDalyTomPetchTomSchollWarrenKumariWesleySteehouwerWillHargraveWim Henderickx


Reuseofthisslidedeckispermittedandencouraged!


Presentationcreatedby:

[email protected]@greg_hankins

[email protected]@JobSnijders

Bonusslides


TheScienceBehindShuttingDownBGPSessions

• AvoidingdisruptionsduringmaintenanceoperationsonBGPsessions:https://inl.info.ucl.ac.be/system/files/ucl-ft-bgp-shutdown-inl.pdf (August2008)

• RequirementsfortheGracefulShutdownofBGPSessionshttps://tools.ietf.org/html/rfc6198 (April2011)


recent bgp innovations for operational challenges · 2019. 10. 1. · recent bgp innovations for...

Documents