8 [france-ix] technical (simon) · 2018-09-24 · france-ix general meeting september 2018 2...
TRANSCRIPT
Technical reportSimon MUYAL
France-IX General Meeting September 20181
Agenda
Technical key numbers
Paris & Marseille infrastructure upgrades
Zoom on outages
Securing route servers
Automating connection process
France-IX General Meeting September 20182
These achievements aretheresult ofthework donebythetechnical team
AlexandreArnaudAnastasiaBoulbabaMikaelPierrePierre-MaloThierryVittorio
Technical key numbers
France-IX General Meeting September 20183
New ports connected during last yearPeriod: From July 2017 to June 2018
France-IX General Meeting September 20184
Global1G +24
10G +62
100G +7
Global +93
Troubleshooting Tickets (NOC)
France-IX General Meeting September 20185
GlobalQ3-2017 54
Q4-2017 92
Q1-2018 75
Q2-2018 20
Global 241
Backbone infrastructure upgrade
Enabling Route server filtering
Other recurring requests• MAC address change• Portal credentials
Resp. time: 4.7/5Web portal: 4.2/5
Network AvailabilityNetwork availability : from july 2017 to june 2018
France-IX General Meeting September 20186
Marseille = 100%no outage
Paris = 99.995%4 outages (detailed later)
Paris infrastructure upgradesOptical infrastructureTelehouse 2Equinix PA6
France-IX General Meeting September 20187
Paris optical infrastructure upgrade8 Coriant Groove G30 deployed
Initial capacity deployed per link: 400G
PoPs upgraded:• Interxion PAR5• Telehouse 2• Interxion PAR2• Equinix PA6
France-IX General Meeting September 20188
SCHEMAavecliens400Gupgradé
400Glink
Paris optical infrastructure upgrade
Very easy to add 200G or 400G of capacity between 2 PoPs
Feedback after 12 months• No issue impacting members• Some python scripts to collect monitoring
information via APIs
France-IX General Meeting September 20189
Paris IP infrastructure upgrades: TH2Starting with core PoPs:
Interxion PAR5 : Done in July 2017
Telehouse2 : Done in November 2017
2 chassis with ~100 customers each one à 2 maintenancesHalf of our customers, half of our traffic…
France-IX General Meeting September 201810
Paris IP infrastructure upgrades: TH2
France-IX General Meeting September 201811
From Juniper EX9200 to Extreme SLX9850-8
Reminder
8 slots36 x 100G LC72 x 1G/10G LC
Paris IP infrastructure upgrades: TH2
Before migration: 2-3 months
• Intensive tests • Scripts to adapt configurations from Juniper to Extreme,
using our Information System• Power increase per rack• Interconnecting existing and new platforms to minimize
the downtime
France-IX General Meeting September 201812
Paris IP infrastructure upgrades: TH2During migration: 02:00 à 08:00am
Moving customers and monitoring in parallelMinimizing downtime per member : 5-10minutes in average
After migration: during some daysDouble checking using our tools:
• Observium: Global statistics, power/fan alerts, etc• Icinga: Customer IPs monitoring• BUM traffic sniffers: same amount of BUM traffic• QoS probes: same delays• BGP sessions with route servers: Always a good indicator
France-IX General Meeting September 201813
Paris IP infrastructure upgrades: TH2200 customers moved from Juniper EX9200 to Extreme SLX-9850
France-IX General Meeting September 201814
PA6 upgrade – July 2018
Increase of 100G requests
~ 20 members/30 ports to migrate
Extreme SLX 9850-8 deployed
France-IX General Meeting September 201815
BaiePA6aprèsmigration
Marseille infrastructure upgradesNew PoP MRS2Additional 100G LCRoute servers upgraded
France-IX General Meeting September 201816
Interxion MRS2 – May 2018
France-IX PoP in the new DC Interxion MRS2
Juniper EX9200 installed• 1G/10G/100G ports available
Interconnected to InterxionMRS1 with dual redundant path
France-IX General Meeting September 201817
BaiePA6aprèsmigration
Interxion MRS2 – May 2018Backbone capacity: 100G between Interxion MRS1 and MRS2, based on passive DWDM MUXes
Easy to evolve to 200G capacity
Possibility to install DWDM active equipment if traffic grows (400G)
France-IX General Meeting September 201818
BaiePA6aprèsmigration
BaieMRS2photo
Marseille infrastructure upgrades
In term of traffic, InterxionMRS1 is our 3rd PoP
France-IX General Meeting September 201819
BaieMRS2photo
1st2nd3rd
TH2PAR2
MRS1
among 12 PoPs
LCs installed to address 100G
customer requests @Interxion MRS1
Physical servers hosting route servers have been
replaced @Interxion MRS1 and
@Jaguar MRS01
Infrastructure upgrades: future worksSolution for edge PoPsInterconnecting Paris and Marseille
France-IX General Meeting September 201820
Edge : finding a solution for dense 100G PoPs
Reminder• Juniper EX9214 deployed in some edge PoPs• Not enough dense in terms of 100G ports and expensive
Solution found in 2017 with Brocade (SLX-9850 and SLX-9540)• But Extreme is not able to maintain pricing conditions obtained previously
Oct-Nov 2018 : Tender to find a box to replace progressively EX9214 when 100G needs increase
France-IX General Meeting September 201821
BaieMRS2photo
Interconnecting Paris and Marseille
Following a recent survey, members are asking for a connection between Paris and Marseille platforms
On the technical side, we will compare and select operators providing 100G waves between Paris and Marseille
We are currently working on how to provide this interconnection to our members
Available beginning of 2019France-IX General Meeting September 2018
22
BaieMRS2photo
Zoom on outages08/2018 – proxy ARP09/2018 – SLX platform
France-IX General Meeting September 201823
Outage : Proxy ARP
Proxy ARP configured on a customer portA member started replying to ”some” ARP requests, giving its MAC address in the reply…
A similar issue occurred 3 years ago
Our tools didn’t detect immediately the issue because the member replies only when the MAC address was not in its cache…
France-IX General Meeting September 201824
BaieMRS2photo
Outage : Proxy ARP: Detecting faster
In the quarantine VLAN, scan all the France-IX LAN IP range, and not only few IPs
Analyse BUM sniffers in real time• Ongoing work• Interesting not only for proxy ARP issue
Proactive: DAI: Dynamic ARP inspection• Analyse ARP traffic and filter ARP replies when they didn’t match
well known static entries (IP <-> MAC)• Available on Extreme SLX platform• Already tested in our lab, waiting for the vendor feedback
France-IX General Meeting September 201825
BaieMRS2photo
Outage : Extreme SLX platform instabilities
Extreme SLX-9850-8, platform stable since September 2017, 4 chassis deployed, connecting 250 customers
• 2 issues encountered over the last 12 months• February 2018 : Reload of one chassis at TH2: 9 minutes
• Combination of several issues• a lack of memory solved in the latest firmware installed• Now: Able to monitor memory and prevent this type of issue
• September 2018 : Reload of 2 100G LC during a logs collection: 8 minutes
• Command identified and documented on our side• Waiting feedback from vendor
France-IX General Meeting September 201826
BaieMRS2photo
Securing route serversApplying strict filtering based on ROA/RPKI and IRR
France-IX General Meeting September 201827
Securing route servers: Story
France-IX started securing route servers 18months agoBGP communities used to tag routes
• IRR not found• ROA invalid
Done using a combination of our DB, bird, some scripts and tools like bgpq3, NTT DB
Every member was able to filter by himself… but this is not enough to have a secure Internet Exchange...
France-IX General Meeting September 201828
51706:65012 = Prefix has ROA status: VALID51706:65022 = Prefix has ROA status: INVALID51706:65023 = Prefix has ROA status: UNKNOWN51706:65011 = Prefix is present in an AS's announced AS/AS-SET51706:65021 = Prefix is not present in an AS's announced AS/AS-SET
Securing route servers: Story
Sept 2017 : Following GM-2017, interesting discussion on tech-ML about applying strict filtering, based on IRR and ROA tags
Oct 2017 : Survey launched to have a better feedback, and not only few people expressing their opinion
Nov 2017: Survey results:• 110 members expressed their opinion!!• 73% in favour of a strict filtering based on ROA and IRR tags
Calendar for applying strict filtering:• Initially announced for January 2018• Applied in February 2018
France-IX General Meeting September 201829
Securing route servers: issues
• Technically, very simple!• “IRR not found” and “ROA invalid” BGP communities
were already applied on routes present in the RS…• Just a new rule in bird to filter these routes by default...
But...
• In nov 2017, 30% of members using RS had at least 1 “ROA invalid” or “IRR not found” route...
• Some announcements made on MLs
France-IX General Meeting September 201830
BaieMRS2photo
Securing route servers: issues
Members having 100% of their routes tagged as invalid:• Invalid AS-SET in peeringdb for example
Members having some of their routes filtered:• AS-SET not up-to-date• Some route objects missing in the IRR
It took some time to reach a situation where members updated their IRR objects
France-IX General Meeting September 201831
BaieMRS2photo
Securing route servers: Giving visibility
A dedicated looking glass to check route validityhttps://lg.franceix.net/welcome/RS1+RS2/ipv4
Documentation describinghow filtering is donehow we consider a route is invalidhttps://www.franceix.net/en/technical/france-ix-route-servers/
à Helping membersà Saving a lot of time for support and troubleshooting
France-IX General Meeting September 201832
Securing route servers: D Day
No main issue encountered
No traffic loss
A couple of members reacted to previous emails to update their route objects
France-IX General Meeting September 201833
BaieMRS2photo
Securing route servers: D Day
Paris7000 IPv4 routes filtered among 100 0004000 IPv6 routes filtered in Paris 28 000
France-IX General Meeting September 201834
BaieMRS2photo
Automating connection process
France-IX General Meeting September 201835
Automating connection process: DONE
Reminder: Our information system based on netbox
• Automating LOA• patch panel integration: Very long to check with DCs the
real status of a patch panel position...• Automating welcome mails
• Automating config checks
France-IX General Meeting September 201836
BaieMRS2photo
Automating connection process: Ongoing
50% of new ports delivered are related to upgrades
• Working on an easy way for members to start the upgrade process and save time
• Rewriting tools.franceix.net to provide additional information
• Survey: One of the points to improve• Technical information (IPs, ports, MACs, patchpannels, RS status
with ROA/IRR filtering)• Commercial view: services, billing, ...• Available beginning of 2019
France-IX General Meeting September 201837
BaieMRS2photo
THANK YOU FOR YOUR ATTENTION
France-IX General Meeting September 201838
+33 (0)1 70 61 97 72
twitter.com/ixpfranceix
facebook.com/ixpfranceix
youtube.com/user/TheFranceIX