best practices for hkix peering · best practices for hkix peering isp symposium 2017 kenneth chan...
TRANSCRIPT
BestPracticesforHKIXPeeringISPSymposium2017
KennethCHANTeamLead,HKIX
www.hkix.net18 Dec2017
WhatisHKIX?
• EstablishedinApr1995,HongKongInterneteXchange(HKIX) isthemainlayer-2InterneteXchangePoint(IXP)inHongKongwherevariousautonomoussystemsinterconnectwithoneanotherandexchangetraffic
• HKIXisnowownedandoperatedbytheHongKongInterneteXchangeLimited(awholly-ownedsubsidiaryofTheChineseUniversityofHongKongFoundationLimited)incollaborationwithInformationTechnologyServicesCentre ofTheChineseUniversityofHongKong
• HKIXservesbothcommercialnetworksandR&Enetworks• Theoriginalgoalistokeepintra-HongKongtrafficwithin
HongKong
HelpKeepIntra-AsiaTrafficwithinAsia
• WehavealmostalltheHongKongnetworks• So,wecanattractparticipantsfromMainlandChina,Taiwan,
Korea,Japan,Singapore,Malaysia,Thailand,Indonesia,Philippines,Vietnam,IndiaandotherAsiancountries
• Wenowhavemorenon-HKroutesthanHKroutes• Wedohelpkeepintra-AsiatrafficwithinAsia• Intermsofnetworklatency,HongKongisagoodcentral
locationinAsia• HKIXdoeshelpHKmaintainasoneoftheInternethubsin
Asia• HKIXsupportsbothdomesticandinternationaltraffic
ISP DISP A ISP B ISP C
Routes of ISP A
Routes of All ISPs in HKIX
Routes of ISP B
Routes of ISP C
Routes of ISP D
Routes of All ISPs in HKIX
Routes of All ISPs in HKIX
Routes of All ISPs in HKIX
MLPARoute
Servers
Routes of All ISPs in HKIX
Routes from All ISPs Switched Ethernet
HKIXModel—MLPAoverLayer2+BLPA
• MLPA traffic exchanged directly over layer 2 without going through MLPA Route Server
• BLPA over layer 2 without involvement of MLPA Route Server
• Supports both IPv4 and IPv6 over the same layer 2 infrastructure
NewHKIXDual-CoreTwo-TierSpine-and-LeafArchitecture
For2014andBeyondHKIX1CoreSite@CUHK HKIX1bCoreSite@CUHK
CoreSwitch@HKIX1
CoreSwitch
@HKIX1b
AccessSwitch(es)@HKIX2
AccessSwitches@HKIX1
AccessSwitches@HKIX1b
AccessSwitch
@HKIX-R&E
------(<2km)------
n x100GE/10GEInter-Switch
Links
n x100GE/10GEInter-Switch
Links
ISP1 ISP2 ISP3 ISP4 ISP5 ISP6 ISP7
CoreSwitch@HKIX1
CoreSwitch
@HKIX1b
AccessSwitch(es)@HKIXm
AccessSwitch(es)@HKIXn
100GE/10GE/GELinks
100GE/10GE/GELinks
HKIXTrafficin2007
HKIXTrafficin2010
HKIXTrafficin2013
HKIXTrafficin2016
HKIXToday• SupportsbothMLPA(MultilateralPeering)andBLPA(BilateralPeering)overlayer2
• SupportsIPv4/IPv6dual-stack• Moreandmorenon-HKparticipants• 270+autonomoussystemsconnected• 500+connectionsintotal
– 20100GE, 300+ 10GE& 170+ GE
• 960+Gbps(5-min)totaltrafficatpeak• AnnualTrafficGrowth~30%
HKIXTrafficDailyGraph(5-minaverage)
HKIXTrafficYearlyGraph(1-dayaverage)
AdvantagesofHKIX• Location
– HongKongisagoodcentrallocationinAsia~50mstoTokyoand~30mstoSingapore
• Neutral– Treatallpartnersequal,bigorsmall– NeutralamongISPs/telcos /localloopproviders/datacenters/
contentproviders/cloudservicesproviders• Trustable
– Treatallpartnersfairandconsistent– Respectbusinesssecretsofeverypartner/participant
• HighPerformance– Nointernalperformancebottleneck,nointernalpacketloss
• NotforProfit– Chargingmainlyforequipmentupgradeandlong-term
sustainability,notforprofit-making
100GConnectionsatHKIX
3
5
7 7 7
9 9
1112
14 14
17
1920
0
5
10
15
20
25HKIX100GPortsConnected(2016NOV- 2017DEC)
100GEYear-Month
Num
bero
fCon
nections
100GParticipantsatHKIX
• Akamai• Amazon• ChinaMobileInternational• CloudFlare• Facebook• Google• HongKongBroadbandNetwork• HurricaneElectric• Tencent• TVB• Yahoo!
HKIXSatelliteSitesHongKong,08Feb2017HKIXannouncesthat3newsatellitesiteswillbeestablishedincollaborationwith3commercialdatacentreswhichprovidecolocationservicesaswellaseasyconnectionstoHKIX.
SatelliteSite
SatelliteSiteCollaborator District PortsSupported Status
HKIX2 CITICTelecomInternational Kwai Chung GE/10GE ReadyforService
HKIX3 SUNeVision /iAdvantage FoTan GE/10GE/100GE ReadyforService28Feb2017
HKIX4 NTTComAsia Tseung KwanO GE/10GE/100GE ReadyforService19Jun2017
HKIX5 KDDI /Telehouse/HKCOLO.net
Tseung KwanO GE/10GE/100GE ReadyforService24Mar2017
• ForconnectionstoHKIXatSatelliteSites,specialconnectionchargeswillbechargedbyrelevantoperators,inadditiontotheportchargeschargedbyHKIX.
• ForHKIXparticipantsnotco-locatedatHKIXsatellitesites,theycanstillconnecttoanyofthetwoHKIXcoresites,i.e.HKIX1andHKIX1bsitesbylocalloopsvialocalloopproviders.
SetupMultipleHKIXSatelliteSites
• AllowparticipantstoconnecttoHKIXmoreeasilyatlowercost fromthosesatellitesitesinHongKong
• OpentocommercialdatacentresinHKwhichfulfilminimumrequirementssoastomaintainneutralitywhichisthekeysuccessfactorofHKIX
• Createawin-winsituationwithsatellitesitecollaborators• TobenamedHKIX2/3/4/5/6/etc
Latestupdates:– HKIX2hasbeenmigratedfromoldmodeltoHKIXSatelliteSite– HKIX3/4/5arenewSatelliteSitesandtheyareReadyforService now
• HKIX1 andHKIX1b (thetwoHKIXcoresiteslocatedwithinCUHKCampus)willcontinuetoserveparticipantsdirectly
HKIX-R&ENode−SupportforNationalR&ENetworksinHongKong• HKIXhelpsthoseR&ENetworksinterconnectamongthemselvesandwith
commercialnetworks withoutrestrictionsviaHKIX-R&EswitchatMEGA-i
• ThemainpurposeistofacilitatethoseNationalR&ENetworkshavingpresenceinHongKongtodointerconnectionsamongthemselves*and* dopeeringwithcommercialnetworks atHKIXmoreeasilyandatalowercost.
• Startedin2008• LocatedinMEGA-iAdvantage• ForResearchandEducationNetworks(R&E)only• SupportGE/10GE/100GE TrunkPorts• SupportPoint-to-pointVLANsforR&Enetworks
– Forprivateinterconnectionsamongany2R&Enetworks– JumboFramesupport
• FiberCrossConnecttobeprovidedbyR&Enetworks• 7x24NOCsupport• OperatebyHKIXwithaNexus7700switchatMEGA-i
HKIX-R&ENodeatMEGA-i
HKIX-R&E
10GE
HKIX
China
270+CommercialNetworks
GE
100GE
10GE
10GE
10GE
10GE
Taiwan
10GE10GE
Taiwan
Korea
Japan
Korea
Singapore
Philippine
China
GENordics
CERNET
CSTNET
APANJP/NICT/JGN-X
KISTI/KREONET2
NIA/KOREN
ASTI/PREGINET
NUS
ASGCNET
ASNET
NORDUnet 20GE
GNA- ABlueprintforGlobalR&ENetworkArchitecture
http://gna-re.net• TheGlobalNetworkArchitectureprogram(GNA)isaninternationalcollaborationbetweennationalresearchandeducation(R&E)networks
• ThediscussionsinsidetheGNAgrouphaveledtoaglobalnetworkarchitecturemodelthatconsistsofapowerfulintercontinentaltransmissionsubstrate,consistingof:– GlobalOpenExchangePoints(GXPs)– High-bandwidthtransmissionpipes(runningbetweenGXPs)forsharing
GNA – artist’s impressionCredit – Mian Usman (DANTE)
PlannedWorksfor2017/18
• ImprovedStability– BetterControlofProxyARP(DONE)– L2ControlonHKIXpeeringLAN(DONE)
• ImprovedServices– SetupSatelliteSitesinmultiplecommercialDataCentre(DONE)– SetupportalforHKIXparticipants(2018Q1)– True24x7NOC(DONE)– Improveafter-hoursupport(DONE)– MoreadvancedRouteServerfeatures(2018Q1)
• ImprovedSecurity– ISO27001(2018Q2)– BettersupportforDDoSMitigation(DONE)
BetterControlofProxyARP
– AutomaticDetectionofProxyARP(implemented)• BasedonduplicatedIPv4ARPentrieslearnedonHKIXRouteServers
– AutomaticshutdownswitchportofHKIXpeercausingProxyARP(willbeimplementedsoon)
– EmailnotificationtoNOCofHKIXpeercausingProxyARP(willbeimplementedsoon)
BetterControlofProxyARP
– Recommendation:• DisableProxyARPCOMPLETELY!!• NorestrictedorunrestrictedProxyARP
– CiscoIOS:• Configurationatinterface:
– noip proxy-arp• Verification:
– showip interface|includeProxyARP– “ProxyARPisdisabled”
– JuniperJUNOS:• ProxyARPisnotenabledbydefault• SodoNOT configurerestrictedorunrestrictedmodeProxyARP
L2ControlforHKIXPeeringLAN
– TrafficAllowedinHKIXPeeringLAN:• EthernetTypes
– 0x0800- IPv4– 0x0806- ARP– 0x86dd- IPv6
• UnicastOnly– NomulticastorbroadcastexceptARPbroadcast
• PortSecurityAlwaysOn– OneMACaddressoneport
AdvancedRouteServerFeatureFeature BGPStandardCommunity
Sendprefixtoall 4635:4635
Sendprefixto$Peer-ASonly 4635:$Peer-AS
Donotsendprefixtoall 0:4635
Do notsendprefixto$Peer-AS 0:$Peer-AS
- TargetforQ1of2018- Support2-byteASnumbersonly- DefaultsendingprefixtoallifnoBGPcommunityistagged
SupportofBlackholing forAnti-DDoSonHKIXRouteServers
HKIXrouteserverssupportRemoteTriggeredBlackHoleFiltering(RTBH)forannouncementofblack-holefiltering
http://www.hkix.net/hkix/anti-ddos.htm
No.ofASNsParticipated:40
Howitworks?• Thevictim’saddressmustbeincludedintheparticipantfilterontheHKIXrouteserversforBGP
announcement• Participanttagthe/32prefixwith4635:666 foritscustomer• HKIXrouteserverssettheprefixwithnexthop123.255.90.66• OtherHKIXparticipantsacceptthe/32prefixandsetthenexthopaddressfor123.255.90.66tonull
ExpectedResults:• Onlythevictim(/32)willbeunreachableviaHKIXnetworkwhilesavingtheothers• TheDDoStrafficwillbeblack-holedatthesideoftheparticipatingrouterswhichareclosertothe
DDoStrafficsources
SupportofBlackholing forAnti-DDoSonHKIXRouteServers(BEFORE)
SupportofBlackholing forAnti-DDoSonHKIXRouteServers(AFTER)
SupportofHidingAS4635fromHKIXRouteServers
• HidingAS4635(ASNofHKIX)ontheASPathintheBGProutesreceivedfromHKIXrouteservers
• SupportbothIPv4and/orIPv6
HKIXParticipantshouldproceedthefollowingsteps:1. DisableBGPEnforcetheFirstAutonomousSystemPathonyour
HKIXpeeringrouter- SampleconfigurationforCiscorouters:
Router(config)#routerbgp <Your-ASN>Router(config-router)#nobgp enforce-first-as
2. NotifyHKIXforhidingAS4635intheBGProutes3. HKIXwillhidetheAS4635ontheASPathfortheIPv4and/orIPv6
routessendingfromHKIXrouteserverstoyourHKIXpeering
PortalforHKIXParticipants
– https://portal.hkix.net– Functions:
• ChangePortSecurity• MRTGStatistics
– Physicalport– LAGport– AggregatedperCustomer
• ScheduleMaintenanceWindow
– [email protected] forpilottestingofHKIXPortal
PortalforHKIXParticipants
• LoginPage(URL:https://portal.hkix.net/)
HKIXPortal– PortSecurity
• Changeportsecurity
HKIXPortal– MRTGStatistics
• Reviewanindividualstatistics/HKIXtotalstatistics
HKIXPortal- MaintenanceWindow
• ScheduleMaintenanceWindow
24x7HKIXNOC
– FulloperationstartingQ1of2017– [email protected] forsecurityoroperationalrelatedmatters
– KeepyourcontactpointatHKIXupdatedforsecurityincidentreporting
OtherOperationalTips
HKIXParticipantsSHOULDNOT:– PerformtestingorloopingonHKIXnetworks– Announcefull/defaultroutetoHKIXrouteservers– AdvertiseHKIXpeeringLANtoothernetworks– Forwardlink-localprotocolstoHKIXPeeringLAN
• IRDP• ICMPredirects• IEEE802SpanningTree• Vendorproprietaryprotocolssuchasdiscoveryprotocols:CDP,EDP• VLAN/Trunkprotocols:VTP,DTP• Interiorroutingprotocolbroadcasts(e.g.OSPF,ISIS,IGRP,EIGRP)• BOOTP/DHCP• PIM-SM• PIM-DM• DVMRP• ICMPv6ND-RA• UDLD• L2Keepalives
OtherOperationalTips
HKIXParticipantsSHOULDDO:– MakesureproxyARPisdisabled– EstablishBGPMLPApeeringwithBOTH HKIXrouteservers– NotifyHKIXNOCforschedulemaintenanceinadvancesothatwewillnottreatyourBGPsessiondownasfailure
– Monitorthegrowthofnumberofprefixesfromourrouteserversandadjustyourmaxprefixsettingaccordingly
– Monitortheutilizationofyourlinkscloselyanddoupgradebeforetheyarefull
– Doyourownroute/route6/as-setobjectsonIRRDBandkeepthemup-to-date
– DoupdateyourcontactandpeeringinfoinPeeringDB
ThankYou!
Forenquiries,[email protected]