best practices for hkix peering · best practices for hkix peering isp symposium 2017 kenneth chan...

BestPracticesforHKIXPeeringISPSymposium2017

KennethCHANTeamLead,HKIX

www.hkix.net18 Dec2017

WhatisHKIX?

• EstablishedinApr1995,HongKongInterneteXchange(HKIX) isthemainlayer-2InterneteXchangePoint(IXP)inHongKongwherevariousautonomoussystemsinterconnectwithoneanotherandexchangetraffic

• HKIXisnowownedandoperatedbytheHongKongInterneteXchangeLimited(awholly-ownedsubsidiaryofTheChineseUniversityofHongKongFoundationLimited)incollaborationwithInformationTechnologyServicesCentre ofTheChineseUniversityofHongKong

• HKIXservesbothcommercialnetworksandR&Enetworks• Theoriginalgoalistokeepintra-HongKongtrafficwithin

HongKong

HelpKeepIntra-AsiaTrafficwithinAsia

• WehavealmostalltheHongKongnetworks• So,wecanattractparticipantsfromMainlandChina,Taiwan,

Korea,Japan,Singapore,Malaysia,Thailand,Indonesia,Philippines,Vietnam,IndiaandotherAsiancountries

• Wenowhavemorenon-HKroutesthanHKroutes• Wedohelpkeepintra-AsiatrafficwithinAsia• Intermsofnetworklatency,HongKongisagoodcentral

locationinAsia• HKIXdoeshelpHKmaintainasoneoftheInternethubsin

Asia• HKIXsupportsbothdomesticandinternationaltraffic

ISP DISP A ISP B ISP C

Routes of ISP A

Routes of All ISPs in HKIX

Routes of ISP B

Routes of ISP C

Routes of ISP D

Routes of All ISPs in HKIX

Routes of All ISPs in HKIX

Routes of All ISPs in HKIX

MLPARoute

Servers

Routes of All ISPs in HKIX

Routes from All ISPs Switched Ethernet

HKIXModel—MLPAoverLayer2+BLPA

• MLPA traffic exchanged directly over layer 2 without going through MLPA Route Server

• BLPA over layer 2 without involvement of MLPA Route Server

• Supports both IPv4 and IPv6 over the same layer 2 infrastructure

NewHKIXDual-CoreTwo-TierSpine-and-LeafArchitecture

For2014andBeyondHKIX1CoreSite@CUHK HKIX1bCoreSite@CUHK

CoreSwitch@HKIX1

CoreSwitch

@HKIX1b

AccessSwitch(es)@HKIX2

AccessSwitches@HKIX1

AccessSwitches@HKIX1b

AccessSwitch

@HKIX-R&E

------(<2km)------

n x100GE/10GEInter-Switch

Links

n x100GE/10GEInter-Switch

Links

ISP1 ISP2 ISP3 ISP4 ISP5 ISP6 ISP7

CoreSwitch@HKIX1

CoreSwitch

@HKIX1b

AccessSwitch(es)@HKIXm

AccessSwitch(es)@HKIXn

100GE/10GE/GELinks

100GE/10GE/GELinks

HKIXTrafficin2007

HKIXTrafficin2010

HKIXTrafficin2013

HKIXTrafficin2016

HKIXToday• SupportsbothMLPA(MultilateralPeering)andBLPA(BilateralPeering)overlayer2

• SupportsIPv4/IPv6dual-stack• Moreandmorenon-HKparticipants• 270+autonomoussystemsconnected• 500+connectionsintotal

– 20100GE, 300+ 10GE& 170+ GE

• 960+Gbps(5-min)totaltrafficatpeak• AnnualTrafficGrowth~30%

HKIXTrafficDailyGraph(5-minaverage)

HKIXTrafficYearlyGraph(1-dayaverage)

AdvantagesofHKIX• Location

– HongKongisagoodcentrallocationinAsia~50mstoTokyoand~30mstoSingapore

• Neutral– Treatallpartnersequal,bigorsmall– NeutralamongISPs/telcos /localloopproviders/datacenters/

contentproviders/cloudservicesproviders• Trustable

– Treatallpartnersfairandconsistent– Respectbusinesssecretsofeverypartner/participant

• HighPerformance– Nointernalperformancebottleneck,nointernalpacketloss

• NotforProfit– Chargingmainlyforequipmentupgradeandlong-term

sustainability,notforprofit-making

100GConnectionsatHKIX

3

5

7 7 7

9 9

1112

14 14

17

1920

0

5

10

15

20

25HKIX100GPortsConnected(2016NOV- 2017DEC)

100GEYear-Month

Num

bero

fCon

nections

100GParticipantsatHKIX

• Akamai• Amazon• ChinaMobileInternational• CloudFlare• Facebook• Google• HongKongBroadbandNetwork• HurricaneElectric• Tencent• TVB• Yahoo!

HKIXSatelliteSitesHongKong,08Feb2017HKIXannouncesthat3newsatellitesiteswillbeestablishedincollaborationwith3commercialdatacentreswhichprovidecolocationservicesaswellaseasyconnectionstoHKIX.

SatelliteSite

SatelliteSiteCollaborator District PortsSupported Status

HKIX2 CITICTelecomInternational Kwai Chung GE/10GE ReadyforService

HKIX3 SUNeVision /iAdvantage FoTan GE/10GE/100GE ReadyforService28Feb2017

HKIX4 NTTComAsia Tseung KwanO GE/10GE/100GE ReadyforService19Jun2017

HKIX5 KDDI /Telehouse/HKCOLO.net

Tseung KwanO GE/10GE/100GE ReadyforService24Mar2017

• ForconnectionstoHKIXatSatelliteSites,specialconnectionchargeswillbechargedbyrelevantoperators,inadditiontotheportchargeschargedbyHKIX.

• ForHKIXparticipantsnotco-locatedatHKIXsatellitesites,theycanstillconnecttoanyofthetwoHKIXcoresites,i.e.HKIX1andHKIX1bsitesbylocalloopsvialocalloopproviders.

SetupMultipleHKIXSatelliteSites

• AllowparticipantstoconnecttoHKIXmoreeasilyatlowercost fromthosesatellitesitesinHongKong

• OpentocommercialdatacentresinHKwhichfulfilminimumrequirementssoastomaintainneutralitywhichisthekeysuccessfactorofHKIX

• Createawin-winsituationwithsatellitesitecollaborators• TobenamedHKIX2/3/4/5/6/etc

Latestupdates:– HKIX2hasbeenmigratedfromoldmodeltoHKIXSatelliteSite– HKIX3/4/5arenewSatelliteSitesandtheyareReadyforService now

• HKIX1 andHKIX1b (thetwoHKIXcoresiteslocatedwithinCUHKCampus)willcontinuetoserveparticipantsdirectly

HKIX-R&ENode−SupportforNationalR&ENetworksinHongKong• HKIXhelpsthoseR&ENetworksinterconnectamongthemselvesandwith

commercialnetworks withoutrestrictionsviaHKIX-R&EswitchatMEGA-i

• ThemainpurposeistofacilitatethoseNationalR&ENetworkshavingpresenceinHongKongtodointerconnectionsamongthemselves*and* dopeeringwithcommercialnetworks atHKIXmoreeasilyandatalowercost.

• Startedin2008• LocatedinMEGA-iAdvantage• ForResearchandEducationNetworks(R&E)only• SupportGE/10GE/100GE TrunkPorts• SupportPoint-to-pointVLANsforR&Enetworks

– Forprivateinterconnectionsamongany2R&Enetworks– JumboFramesupport

• FiberCrossConnecttobeprovidedbyR&Enetworks• 7x24NOCsupport• OperatebyHKIXwithaNexus7700switchatMEGA-i

HKIX-R&ENodeatMEGA-i

HKIX-R&E

10GE

HKIX

China

270+CommercialNetworks

GE

100GE

10GE

10GE

10GE

10GE

Taiwan

10GE10GE

Taiwan

Korea

Japan

Korea

Singapore

Philippine

China

GENordics

CERNET

CSTNET

APANJP/NICT/JGN-X

KISTI/KREONET2

NIA/KOREN

ASTI/PREGINET

NUS

ASGCNET

ASNET

NORDUnet 20GE

GNA- ABlueprintforGlobalR&ENetworkArchitecture

http://gna-re.net• TheGlobalNetworkArchitectureprogram(GNA)isaninternationalcollaborationbetweennationalresearchandeducation(R&E)networks

• ThediscussionsinsidetheGNAgrouphaveledtoaglobalnetworkarchitecturemodelthatconsistsofapowerfulintercontinentaltransmissionsubstrate,consistingof:– GlobalOpenExchangePoints(GXPs)– High-bandwidthtransmissionpipes(runningbetweenGXPs)forsharing

GNA – artist’s impressionCredit – Mian Usman (DANTE)

PlannedWorksfor2017/18

• ImprovedStability– BetterControlofProxyARP(DONE)– L2ControlonHKIXpeeringLAN(DONE)

• ImprovedServices– SetupSatelliteSitesinmultiplecommercialDataCentre(DONE)– SetupportalforHKIXparticipants(2018Q1)– True24x7NOC(DONE)– Improveafter-hoursupport(DONE)– MoreadvancedRouteServerfeatures(2018Q1)

• ImprovedSecurity– ISO27001(2018Q2)– BettersupportforDDoSMitigation(DONE)

BetterControlofProxyARP

– AutomaticDetectionofProxyARP(implemented)• BasedonduplicatedIPv4ARPentrieslearnedonHKIXRouteServers

– AutomaticshutdownswitchportofHKIXpeercausingProxyARP(willbeimplementedsoon)

– EmailnotificationtoNOCofHKIXpeercausingProxyARP(willbeimplementedsoon)

BetterControlofProxyARP

– Recommendation:• DisableProxyARPCOMPLETELY!!• NorestrictedorunrestrictedProxyARP

– CiscoIOS:• Configurationatinterface:

– noip proxy-arp• Verification:

– showip interface|includeProxyARP– “ProxyARPisdisabled”

– JuniperJUNOS:• ProxyARPisnotenabledbydefault• SodoNOT configurerestrictedorunrestrictedmodeProxyARP

L2ControlforHKIXPeeringLAN

– TrafficAllowedinHKIXPeeringLAN:• EthernetTypes

– 0x0800- IPv4– 0x0806- ARP– 0x86dd- IPv6

• UnicastOnly– NomulticastorbroadcastexceptARPbroadcast

• PortSecurityAlwaysOn– OneMACaddressoneport

AdvancedRouteServerFeatureFeature BGPStandardCommunity

Sendprefixtoall 4635:4635

Sendprefixto$Peer-ASonly 4635:$Peer-AS

Donotsendprefixtoall 0:4635

Do notsendprefixto$Peer-AS 0:$Peer-AS

- TargetforQ1of2018- Support2-byteASnumbersonly- DefaultsendingprefixtoallifnoBGPcommunityistagged

SupportofBlackholing forAnti-DDoSonHKIXRouteServers

HKIXrouteserverssupportRemoteTriggeredBlackHoleFiltering(RTBH)forannouncementofblack-holefiltering

http://www.hkix.net/hkix/anti-ddos.htm

No.ofASNsParticipated:40

Howitworks?• Thevictim’saddressmustbeincludedintheparticipantfilterontheHKIXrouteserversforBGP

announcement• Participanttagthe/32prefixwith4635:666 foritscustomer• HKIXrouteserverssettheprefixwithnexthop123.255.90.66• OtherHKIXparticipantsacceptthe/32prefixandsetthenexthopaddressfor123.255.90.66tonull

ExpectedResults:• Onlythevictim(/32)willbeunreachableviaHKIXnetworkwhilesavingtheothers• TheDDoStrafficwillbeblack-holedatthesideoftheparticipatingrouterswhichareclosertothe

DDoStrafficsources

SupportofBlackholing forAnti-DDoSonHKIXRouteServers(BEFORE)

SupportofBlackholing forAnti-DDoSonHKIXRouteServers(AFTER)

SupportofHidingAS4635fromHKIXRouteServers

• HidingAS4635(ASNofHKIX)ontheASPathintheBGProutesreceivedfromHKIXrouteservers

• SupportbothIPv4and/orIPv6

HKIXParticipantshouldproceedthefollowingsteps:1. DisableBGPEnforcetheFirstAutonomousSystemPathonyour

HKIXpeeringrouter- SampleconfigurationforCiscorouters:

Router(config)#routerbgp <Your-ASN>Router(config-router)#nobgp enforce-first-as

2. NotifyHKIXforhidingAS4635intheBGProutes3. HKIXwillhidetheAS4635ontheASPathfortheIPv4and/orIPv6

routessendingfromHKIXrouteserverstoyourHKIXpeering

PortalforHKIXParticipants

– https://portal.hkix.net– Functions:

• ChangePortSecurity• MRTGStatistics

– Physicalport– LAGport– AggregatedperCustomer

• ScheduleMaintenanceWindow

– ContactHKIXTeamatprovision@hkix.net forpilottestingofHKIXPortal

PortalforHKIXParticipants

• LoginPage(URL:https://portal.hkix.net/)

HKIXPortal– PortSecurity

• Changeportsecurity

HKIXPortal– MRTGStatistics

• Reviewanindividualstatistics/HKIXtotalstatistics

HKIXPortal- MaintenanceWindow

• ScheduleMaintenanceWindow

24x7HKIXNOC

– FulloperationstartingQ1of2017– Contactusatnoc@hkix.net forsecurityoroperationalrelatedmatters

– KeepyourcontactpointatHKIXupdatedforsecurityincidentreporting

OtherOperationalTips

HKIXParticipantsSHOULDNOT:– PerformtestingorloopingonHKIXnetworks– Announcefull/defaultroutetoHKIXrouteservers– AdvertiseHKIXpeeringLANtoothernetworks– Forwardlink-localprotocolstoHKIXPeeringLAN

• IRDP• ICMPredirects• IEEE802SpanningTree• Vendorproprietaryprotocolssuchasdiscoveryprotocols:CDP,EDP• VLAN/Trunkprotocols:VTP,DTP• Interiorroutingprotocolbroadcasts(e.g.OSPF,ISIS,IGRP,EIGRP)• BOOTP/DHCP• PIM-SM• PIM-DM• DVMRP• ICMPv6ND-RA• UDLD• L2Keepalives

OtherOperationalTips

HKIXParticipantsSHOULDDO:– MakesureproxyARPisdisabled– EstablishBGPMLPApeeringwithBOTH HKIXrouteservers– NotifyHKIXNOCforschedulemaintenanceinadvancesothatwewillnottreatyourBGPsessiondownasfailure

– Monitorthegrowthofnumberofprefixesfromourrouteserversandadjustyourmaxprefixsettingaccordingly

– Monitortheutilizationofyourlinkscloselyanddoupgradebeforetheyarefull

– Doyourownroute/route6/as-setobjectsonIRRDBandkeepthemup-to-date

– DoupdateyourcontactandpeeringinfoinPeeringDB

ThankYou!

Forenquiries,pleasecontactusatinfo@hkix.net