Technical Overview By Simon MUYAL

30th September 2016

Agenda

1

Backbone infrastructure update Network stability and availability Paris and Marseille upgrades

Agenda France-IX Technical update

2

Deployments SLA probes Flow statistics

Next steps for 100G optical transport 100G density on IXP fabric More visibility

2017

Backbone infrastructure update

3

More and more stability Network availability (October 2015 to

September 2016)

Network stability

Marseille = 100%

Paris > 99.99%, one outage

4

Hardware

Network stability

No hardware issues observed

5

Software

No software upgrade needed for Brocade JunOS: 2 upgrades

Network stability

Minor issue related to LAG configuration on 100G interfaces

Major issue for visibility : IPFIX bug à Flow statistics unavailable

6

Decembre 2015 A member started spoofing IPv4 adresses from other members Replying to ARP requests, not destined to its IPv4 address Some members have been impacted by this non voluntary « IP spoofing »

Less than 1hour of outage on the last 12 months

8

Improvements done Additional tests made on quarantine VLAN before moving a port into production (ARP PING scan on LAN IP prefix)

Improvements under consideration

Shutdown « automatically » a member port having spoofed an IP address

Less than 1hour of outage on the last 12 months

9

Paris: + 50%

Traffic growing

10

Marseille: +100%

Reports available every quarter

More visibility

11

Sent to Marseille and Paris member’s lists

Contain only technical information

•  Current and new deployments •  New services •  Outages, Maintenances

Additional information on tools portal

More visibility

12

Weathermap, looking glass, …

Flow statistics again available

Why?

Cleaning PoPs and optimizing operations

13

Beautiful but also…

Reducing risk of « accident » when we perform operations on site

Easy to ask for remote Hands&Eyes

Removing old copper cables Redoing fiber cabling properly

Pre-cabling Avoid to go onsite for every customer connection à save time, money à speed-up new connections : Able to deliver a new

port on the same business day

Cleaning PoPs and optimizing operations

14

15/20

16/20

Telehouse 2 upgrade: New Juniper EX9200 installed

•  More redundancy, additional capacity •  Customers split on 2 devices

Paris infrastructure update

17

TH2 PoPs Edge

1/3

TH2

PAR5 PAR2

PoPs Edge

Iliad DC2 and DC3 upgrade

Paris infrastructure update 2/3

18

Backbone capacity upgrade 200Gbps

•  From 70G to 200Gbps

2 Juniper EX9200 installed •  2 Brocade MLX-8

removed

Fast delivery 100G customer’s interfaces available

Several 1G and 10G line cards installed to address new customer requests

A Juniper EX9200 and DWDM MUXs in stock ready to deploy

Able to deploy a 100G PoP in less than 2 weeks •  New PoP or upgrade existing PoP

Paris infrastructure update 3/3

19

Interxion MRS1 PoP Juniper EX9200 installed

100G ports available @Interxion MRS1 @Jaguar Network MRS01

2x40Gbps of capacity between Interxion and Jaguar PoP

Up to 400Gbps with existing solution

Marseille infrastructure update

20

Deployments

21

SLA probes

Study done to select the most appropriate solution

•  Home made solution vs Turnkey solution

•  Dedicated probes,

one per PoP

•  10Gbps vs 1Gbps

Main measurements: •  Delay •  Jitter •  Frame loss

•  Other KPI like

Reachability •  Per customer •  Per PoP •  Global

22

Study

Physical deployment

Dell R320 with 2x10Gbps interfaces deployed Paris and Marseille (11 PoPs) Connected to the peering LAN à same behaviour than a customer

SLA probes Deployment

23

Operating system: Linux with no virtualisation to optimize performances

Some tests made with KVM à bad results when we try to reach 10Gbps

Ansible used for deploying and maintaining SLA probes

SLA probes Deployment

24

Measurements Scapy library and python scripts used to perform measurements Full-mesh Any-to-Any à Matrix available LAG: how to test all the links belonging to a LAG?

•  Hashing : Different IPs SRC/DST and L4 Ports

SLA probes Measurements

25

Storage Influxdb

•  Time series DB •  Continues queries for aggregation

Visualisation Grafana for internal use Integration on tools.franceix.net soon

•  Data available, at least for members

SLA probes Measurements

26

SLA probes

infrastructure

27

Measurements-FranceIXpeeringLAN

DATAexport–OOBLAN

Underdevelopment

SLA probes Visualisation

28

SLA contract available soon: Possibility to have a contract with SLAs

Data visualization available before end of 2016

SLA probes Next steps

29

•  Useful for members when they don’t have flow statistics tools – Top talkers

•  Useful for global statistics – Compare with SNMP - –  IPv6 statistics – …

Flow statistics

30

Flow export: sFlow on Brocade MLX IPFIX on Juniper EX9200 Sampling rate: 1/4096

•  Enough for MAC SRC/MAC DST accounting

PMACCT for flow collection: Thanks for this great open source tool!!

Flow statistics

31

InfluxDB for Flow storage Same tool than SLA probles Scalable: >300 members à #10 000 flows to store

RabbitMQ for queuing between PMACCT and infuxDB

Useful if there is an issue on InfluxDB or between PMACCT and influxDB APIs

Visualisation: integrated on tools.franceix.net

Flow statistics

32

Flow statistics

infrastructure

33

sFlow

IPFIX

Flow statistics Visualisation

34

Next steps for 2017

35

Currently: Passive muxes and 10G DWDM SFP+ Cost effective solution

•  Passive MUX: very cheap •  10G DWDM SFP+ cheap •  Up to 40 channels à 400Gbps

But…

•  Difficult to deploy and maintain •  Some limits on L3 equipment for aggregating links (LAG) •  Poor optical monitoring (DOM integrated in SFP+

transceiver)

Optical infrastructure renewal

36

100G optical solutions are more affordable

Interesting solution for DC interconnects (low distance less than 30km)

Optical infrastructure renewal

37

Study on going to deploy this type of infrastructure beginning of S2

2017

Goal: Increase 100G port density Around 10 x 100G ports deployed before the end 2016 100G port has to be more cost effective

Objective: Deploy a new IP infrastructure in the core during summer 2017

RFP beginning of 2017 Deployment S2 2017

IP infrastructure renewal

38

Available before end of 2016

BGP BH Next hop MAC address associated to this BH NH This MAC address is filtered on the edge

Possibility to use BGP communities selective BH possible

AntiDDOS Blackholing (BH)

39

Dashboard in tools.franceix.net with useful information

Outages Maintenances Network Availability SLA probes information

New members connected Last upgrades …

More visibility

40

Automate the processes

Minimize the time for connecting a new member

Automation

41

Research Build automated APIs for resellers Build automated APIs for marketplace vendors

Sysadmin

Network engineers

Network technician

àSend me an email smuyal@franceix.net

We are hiring

42

Config management

Technical update – part 2 by Arnaud

43

Logs TACACS

AS57734

Oxidized

Route servers

44/20

Thank you for your attention