six and some best practices for running an ixp
DESCRIPTION
Presented at ISOC's Serbian Open Exchange – IXP Workshop, December 4-5, 2013TRANSCRIPT
Matjaž Straus Istenič, SIX, [email protected]
SIX and some best practices for running an IXPAll that stuff around the switch
1sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
About me
2
2sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
About me
2
2sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
About me
2
Triglav, 2864 m
2sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Agenda• Slovenian Internet Exchange - SIX• all that stuff around the switch• practical examples
– addressing– best practices– configuration examples for the IXP– configuration examples, guidelines and hints for members
2/417
3sreda, 04. december 13
Matjaž Straus Istenič, SEE2, Macedonia, 4/2013
SIX - operated by ARNESsince 1994
photo: http://www.pivo-lasko.si/
4
Slovenia has tradition - not only in breweries
4sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
SIX - the history
5
• started in february 1994- two members
• 1995: two more members• 1996: two more...
– and the big Telecom• 1997 ... 2002: more alternative providers• 2000: second location, interconnect with the first at 2001• 2003: third location (LIX, decommissioned 3/2012)• 2006: first IPv6 at SIX• 2009: new location at Ljubljana Technology Park
5sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
SIX - the forbidden graphs
6
0
3
6
9
12
15
18
21
24
27
30
1994 1996 1998 2000 2002 2004 2006 2008 2010 2012
SIX members
• 26 members• > 50% with 10 G• most with IPv6
6sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
SIX - the IXP technology
7
7sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
SIX - today• L2• two locations• *national• Cisco 4500X• bird route server• IXP manager and portal
8*note: one cross-border link
8sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Stuff around the switch• proper location with many fibre providers
– a building with one single provider is a bad idea• different fibre paths inside of the building• power supplies and grounding• cooling system• physical security• staff, support, remote hands• good and accurate documentation
9
9sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Stuff around the switch (cont.)
• monitoring and alarming• ticketing system• mailing lists• web portal• best current practices and knowledge base• contracts, SLAs, billing, ...
• planning for a collocation/datacenter
10
10sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
The power
11
11sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
The power• allocate up to 20 kW per rack• actual usage 5 kW - 10 kW per rack• dual separate circuit breaker for each rack• power supply redundancy
– dual feed from electrical distribution company– separate dual UPS system N+1 and PDU– diesel generator (redundant)
• cooling equipment is independently dual powered, including chillers
• how much power does datacenter use– monitoring on UPS, on PDU– monitoring total on main branch circuit
• typicaly the load will double in 5 years
12
12sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Cooling• full redundancy of cooling system
– two different power grids– separate piping– chiller redundancy– room units redundancy
• hot/cold isle– reduce air mixing– cold aisle with barriers made of metal, plastic or fiberglass– use blanking panels on the cabinets without servers
• no need for double floor– run network cabling over the top of the cabinets– "in row" cooling
• recommended temperature in cold isle is between 23 - 25 °C• cooling system rating must be 1.3 x IT load rating• make sure that the space will allow for future growth
– for more cooling capacity and redundancy if required• Power usage effectiveness (PUE = Total Facility Power/IT Equipment Power)
– typical PUE is 2.0 or higher
13
13sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011 14
14sreda, 04. december 13
15sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Fire protection• sensing the smoke/fire
16
type ✔ ✗
aspiration sensor
• very sensitive• early warning• single point of electrical
instalation• targeted sensing is possible
• more expensive• air ducting under the ceiling
must be installed
optical sensor• cheaper• can be used as confirmation
for fast aspiration sensors
• less sensitive• each sensor needs its own
cable
16sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Fire protection• extinguishing fire
17
Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!
type active substance ✔ ✗
displacement of air
Inergen- mixture of gases, displaces air with “air” with less oxygen
• totally natural• environmentaly
neutral
• big storage requirements• high pressure (200 or 300 bar)• computer room needs big exhaust vents• big rush of gas at release causes dust and
objects to lift
chemical action
Novec 1230- chemical bonding, cooling
• small storage area• stored as fluid• very small greenhouse
gas footprint
• has some effect on environment• expensive• stored under pressure (40/50 bar)chemical
actionFM200 (phasing out)- chemical bonding
• small storage area• small greenhouse gas
footprint
• being phased out• has some ozone depletion impact• stored under pressure (40/50 bar)
cooling water mist• totally natural• environmentaly
neutral
• water in computer room is not a good idea ;-)• possible condensation on cold surfaces
17sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Fire protection• extinguishing fire
17
Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!
type active substance ✔ ✗
displacement of air
Inergen- mixture of gases, displaces air with “air” with less oxygen
• totally natural• environmentaly
neutral
• big storage requirements• high pressure (200 or 300 bar)• computer room needs big exhaust vents• big rush of gas at release causes dust and
objects to lift
chemical action
Novec 1230- chemical bonding, cooling
• small storage area• stored as fluid• very small greenhouse
gas footprint
• has some effect on environment• expensive• stored under pressure (40/50 bar)chemical
actionFM200 (phasing out)- chemical bonding
• small storage area• small greenhouse gas
footprint
• being phased out• has some ozone depletion impact• stored under pressure (40/50 bar)
cooling water mist• totally natural• environmentaly
neutral
• water in computer room is not a good idea ;-)• possible condensation on cold surfaces
17sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Examples and guidelines• addressing• IXP port configuration• guidelines for members• goodies
18
18sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Examples: addressing• a single subnet taken from independent address space
– member address is assigned per location• address schema at SIX
19
91.220.194.n/24n = n1 = 2..99 at location 1n = n1 + 100 = 102..199 at location 2n = 1, 101 for route-reflectors
2001:7f8:46:0:L:N::<AS>/64L = 0 at location 1L = 1 at location 2N = 0 for a single router, otherwise N = 1, 2, ...AS = member AS in decimalAS = 51988 for route-server- diverse lower 24 bits which
form solicited-node mcast address
19sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Examples: IXP port configuration• access port on Cisco 4500X
20
interface TenGigabitEthernet1/6 description -- member (AS...) -- switchport access vlan <x> switchport mode access switchport nonegotiate switchport port-security load-interval 30 datalink flow monitor FlowMonitor-L2 input storm-control broadcast level 1.00 storm-control action shutdown no cdp enable spanning-tree portfast spanning-tree bpduguard enable service-policy input INPUT-200M-EF!policy-map LIMIT-QUEUE-200 class class-default queue-limit 200!
flow record StandardFlow-L2 match datalink mac source address input match datalink mac destination address input collect interface input collect interface output collect counter bytes long collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last!flow exporter FlowExporter destination <x.y.z.w> vrf mgmtVrf source FastEthernet1 transport udp <port> template data timeout 60!flow monitor FlowMonitor-L2 record StandardFlow-L2 exporter FlowExporter cache timeout active 60!
20sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Examples: IXP port configuration• interconnecting ports
– aggregated to EtherChannel with LACP– maximal MTU
21
interface TenGigabitEthernet1/1 switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 load-interval 30 datalink flow monitor FlowMonitor-L2 input channel-protocol lacp channel-group 48 mode active!interface TenGigabitEthernet1/2 switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 load-interval 30 datalink flow monitor FlowMonitor-L2 input channel-protocol lacp channel-group 48 mode active!
interface Port-channel48 description -- IX-trunk -- switchport switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 bandwidth 10000000 load-interval 30 datalink flow monitor FlowMonitor-L2 input flowcontrol receive on!port-channel load-balance src-dst-ip
21sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Guidelines for members• the bads (proxy ARP, redirects)• access port configuration• BGP
– routing considerations (next-hop, localization)– safety (MD5 authentication)– policy (filtering announcements)
– control received prefixes– control advertised prefixes
– RPKI
22
22sreda, 04. december 13
Presenter Name, Date
Proxy ARP in action• incident at AMS-IX, DE-CIX, ...
23
- proxy ARP enabled
- router has no IP address from the peering LAN- router has a default route
or
- router has a more specific route for the peering LAN
reference: Maksym Tulyuk, Wolfgang Tremmel, reported at RIPE63http://ripe63.ripe.net/presentations/
23sreda, 04. december 13
Presenter Name, Date
ARP hijacking• no RS, full BGP mesh between R2, R3 in R4• normal situation
24source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
R1MAC A
R4MAC D
R3MAC C
R2MAC B
R1 is at AR3 is at CR4 is at D
R1 is at AR2 is at BR4 is at D
R1 is at AR2 is at BR3 is at C
R2 is at BR3 is at CR4 is at D
BGP
BGP
BGP
24sreda, 04. december 13
Presenter Name, Date
ARP hijacking (2/8)
• R1 send bogus ARP replies
25source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
R1MAC A
R4MAC D
R3MAC C
R2MAC B
R1 is at AR3 is at CR4 is at D
R1 is at AR2 is at BR4 is at D
R1 is at AR2 is at BR3 is at C
R2 is at AR3 is at AR4 is at A
R2 is at BR3 is at CR4 is at D
BGP
BGP
BGP
25sreda, 04. december 13
Presenter Name, Date
ARP hijacking (3 /8)
• ARP cache poisioned• BGP down• traffic stops
26source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
R1MAC A
R4MAC D
R3MAC C
R2MAC B
R1 is at AR3 is at AR4 is at A
R1 is at AR2 is at AR4 is at A
R1 is at AR2 is at AR3 is at A
26sreda, 04. december 13
Presenter Name, Date
ARP hijacking (4/8)
• hijacker R1 isolated• ARP caches recover with BGP packets• BGP up• traffic normalizes after a few
minutes
27source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
R4MAC D
R3MAC C
R2MAC B
R3 is at CR4 is at D
R2 is at BR4 is at D
R2 is at BR3 is at C
BGP
BGP
BGP
27sreda, 04. december 13
Presenter Name, Date
What if route-server is being used?
28source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
• RS, partial BGP between RS and R2, R3• normal situation
RSMAC D
R3MAC C
R2MAC B
R1MAC A
R1 is at AR3 is at CRS is at D
R1 is at AR2 is at BRS is at D
R1 is at AR2 is at BR3 is at C
BGP
BGP
R2 is at BR3 is at CRS is at D
28sreda, 04. december 13
Presenter Name, Date 29source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
• R1 send bogus ARP replies
RSMAC D
R3MAC C
R2MAC B
R1MAC A
R1 is at AR3 is at CRS is at D
R1 is at AR2 is at BRS is at D
R1 is at AR2 is at BR3 is at C
BGP
BGP
R2 is at AR3 is at ARS is at A
R2 is at BR3 is at CRS is at D
ARP hijacking (6/8)
29sreda, 04. december 13
Presenter Name, Date
ARP hijacking (7/8)
30source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
• ARP cache poisioned• BGP with RS down• traffic stops
RSMAC D
R2MAC B
R3MAC C
R1MAC A
R1 is at AR3 is at ARS is at A
R1 is at AR2 is at ARS is at A
R1 is at AR2 is at AR3 is at A
30sreda, 04. december 13
Presenter Name, Date
ARP hijacking (8/8)
• hijacker R1 isolated• ARP caches partially recover with BGP packets• BGP up• traffic is being
blackholed– R2 and R3 still have bogus entries
for each other
– outage can last for hours
31source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
RSMAC D
R3MAC C
R2MAC B
R1 je na AR3 je na ARS je na D
R1 je na AR2 je na ARS je na D
R2 je na BR3 je na C
BGP
BGP
R1MAC A
31sreda, 04. december 13
Presenter Name, Date
ARP Sponge• update ARP caches• mitigates unknown unicast
32
from to message
sponge B reply: R3 is at C
sponge B request: where is R3? - tell R3 at C
sponge B request: where is R2? - tell R3 at C
B C reply: R2 is at B
source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net
• spoofed unsolicited ARP reply• spoofed gratuitous ARP query• spoofed ARP request
(dve muve jednim udarcem ;-))
• 3 ARP updatemethods: R3
MAC C
R2MAC B
R3 is at C
R2 is at B
R2 is at BR3 is at Cin case of unknown unicast: - the unknown is here
32sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
Tuđe nećemo, ...
33
• R2 peers with R1 but not with R3
• R2 doesn’t want to receive any traffic from R3
33sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
Tuđe nećemo, ...
33
• R2 peers with R1 but not with R3
• R2 doesn’t want to receive any traffic from R3
33sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
Tuđe nećemo, ...
33
✔ preffered path✗ path to avoid
• R2 peers with R1 but not with R3
• R2 doesn’t want to receive any traffic from R3
33sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
Tuđe nećemo, ...
33
✔ preffered path✗ path to avoidnext-hop self in eBGP✔
• R2 peers with R1 but not with R3
• R2 doesn’t want to receive any traffic from R3
with next-hop self at R1 next-hop for 2001:db8::/48 at R3 is A, not B
33sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
... svoje ne damo!
34
✔ preffered path✗ path to avoid
• R1 receives traffic and sends it back via the same port
• R1 doesn’t want to redirect any traffic to R2
34sreda, 04. december 13
Presenter Name, Date
eBGP
eBGP
R1
R2
R3
A
B
2001:db8::/48
next-
hop(2
001:d
b8::/4
8) = B
next-hop(2001:db8::/48) = A or B?
... svoje ne damo!
34
no ip redirects✔✔ preffered path✗ path to avoid
• R1 receives traffic and sends it back via the same port
• R1 doesn’t want to redirect any traffic to R2
ICMP redirect messages should not be sent
34sreda, 04. december 13
Presenter Name, Date
Unreachables and PMTU discovery
35
9000
1500
ICMP 3/4Packet too big, fragmentation required and DF flag set
35sreda, 04. december 13
Presenter Name, Date
Unreachables and PMTU discovery
35
9000
1500
ICMP 3/4Packet too big, fragmentation required and DF flag set
ICMP unreachables are always sent for IPv4
note:In IPv6, ICMP Packet-too-big message is not an “Unreachable”
35sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Example: member port configuration• turn off anything but IP and ARP
36
! example for Cisco IOS!interface TenGigabitEthernet3/3 ip address x.y.z.w 255.255.255.0 ip access-group IxIncoming in ip access-group IxOutgoing out no ip redirects no ip proxy-arp ipv6 address 2001:.../64 ipv6 enable ipv6 traffic-filter IxIncoming6 in ipv6 traffic-filter IxOutgoing6 out ipv6 nd reachable-time 300000 ipv6 nd ra suppress no ipv6 redirects storm-control broadcast level 1.00 no cdp enable!
– no proxy ARP– no redirects– no vendor proprietaryprotocols like CDP
– no broadcasts– no IPv6 RA– ICMP unreachables areused in PMTU discovery in IPv4
36sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Multiple locations• routing considerations
– localize traffic– minimize traffic between locations
37
IXP LANinterconnect
A A
B B
C D
IXP @location 1 IXP @location 2
37sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Multiple locations• routing considerations
– localize traffic– minimize traffic between locations
37
IXP LANinterconnect
A A
B B
C D
IXP @location 1 IXP @location 2
37sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Multiple locations• routing considerations
– localize traffic– minimize traffic between locations
37
IXP LANinterconnect
A A
B B
C D
IXP @location 1 IXP @location 2
37sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Localization
38
membe
r at o
ne
locati
on
membe
r at tw
o loc
ation
siB
GP
eBGP
eBGPR1
R2
R3
A
B
2001:db8::/48
next-hop(2001:db8::/48) = A
next-hop(2001:db8::/48) = B
next-hop(2001:db8::/48) = R2
38sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Localization
38
membe
r at o
ne
locati
on
membe
r at tw
o loc
ation
siB
GP
eBGP
eBGPR1
R2
R3
A
B
2001:db8::/48
next-hop(2001:db8::/48) = A
next-hop(2001:db8::/48) = B
next-hop(2001:db8::/48) = R2
38sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Localization
38
membe
r at o
ne
locati
on
membe
r at tw
o loc
ation
siB
GP
eBGP
eBGPR1
R2
R3
A
B
2001:db8::/48
next-hop(2001:db8::/48) = A
next-hop(2001:db8::/48) = B
next-hop(2001:db8::/48) = R2
I’m marking my prefixes with community for blue location
I’m marking my prefixes with community for red location
I preffer prefixes with red community
• use BGP communities
38sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Examples: localization• Cisco IOS
39
! router at location 1ip community-list 61 permit 65432:1!route-map AnnounceToIX permit 10 set community 65432:1!route-map AcceptFromIX permit 10 ! this location match community 61route-map AcceptFromIX permit 20 ! other location - worse metric set metric +1!router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX!
! router at location 2ip community-list 62 permit 65432:2!route-map AnnounceToIX permit 10 set community 65432:2!route-map AcceptFromIX permit 10 ! this location match community 62route-map AcceptFromIX permit 20 ! other location - worse metric set metric +1!router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX!
39sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
/* router at location 1 */protocols { bgp { local-as <member-AS>; group Ix { type external; import [ LocalizeTraffic AcceptFromIx ]; export AnnounceToIx; } }}policy-options { policy-statement AcceptFromIx { <member policy at receive> } policy-statement AnnounceToIx { term Localize { then { community set IxLocation1; next term; } } <member policy for announcements> } policy-statement LocalizeTraffic { term LocalTraffic { from community IxLocation1; then next policy; } term OtherTraffic { then { metric add 1; } } } community IxLocation1 members 65432:1;}
Examples: localization• Juniper JUNOS
40
/* router at location 2 */protocols { bgp { local-as <member-AS>; group Ix { type external; import [ LocalizeTraffic AcceptFromIx ]; export AnnounceToIx; } }}policy-options { policy-statement AcceptFromIx { <member policy at receive> } policy-statement AnnounceToIx { term Localize { then { community set IxLocation2; next term; } } <member policy for announcements> } policy-statement LocalizeTraffic { term LocalTraffic { from community IxLocation2; then next policy; } term OtherTraffic { then { metric add 1; } } } community IxLocation2 members 65432:2;}
40sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
BGP• authenticate (secure) BGP session• filter announcements• sanity checks• a must-read
– BGP Operations and Securityhttp://tools.ietf.org/id/draft-jdurand-bgp-security-02.txt
– Internet Exchange Route Server Operationhttp://tools.ietf.org/html/draft-ietf-grow-ix-bgp-route-server-operations-01
41
41sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Example: BGP filters
42
router bgp 65432 template peer-policy Ix6 route-map AcceptFromIx in route-map AnnounceToIx out filter-list 200 out prefix-list FromIx6 in prefix-list ToIx6 out next-hop-self remove-private-as maximum-prefix 1000 send-community! template peer-session Ix6 password <default_key> ttl-security hops 1 update-source Vlan50! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> inherit peer-session Ix6 neighbor <...> password <another_key> neighbor <...> filter-list 100 in...!
• remove your own communities <your-as>:<community>• accept only communities that are meaningful for you• respect “no-export”• do not remove other communities for no reason
• properly mark your prefixes
• limit the number of accepted prefixes(beware of the full routing table!)
• authenticate with MD5• TTL security (optional)
42sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Example: prefix filters
43
router bgp 65432 template peer-policy Ix6 filter-list 200 out prefix-list FromIx6 in prefix-list ToIx6 out ... exit-peer-policy ! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> filter-list 100 in!ipv6 prefix-list FromIx6 seq 5 deny ::/0ipv6 prefix-list FromIx6 seq 10 deny <our-prefix>/32ipv6 prefix-list FromIx6 seq 15 deny <our-prefix>/32 ge 33ipv6 prefix-list FromIx6 seq 15 deny ::/0 ge 48ipv6 prefix-list FromIx6 seq 20 deny 2002::/16 ge 17ipv6 prefix-list FromIx6 seq 99 permit 2000::/3 ge 4!ipv6 prefix-list ToIx6 seq 5 permit <our-prefix>/32ipv6 prefix-list ToIx6 seq 10 permit <customer1>/32ipv6 prefix-list ToIx6 seq 15 permit <customer2>/48...!ip as-path access-list 100 permit ^(65000_)+$ip as-path access-list 100 permit ^(65000_)+.*(65001_)+$ip as-path access-list 100 permit ^(65000_)+.*(65002_)+$!ip as-path access-list 200 permit ^$ip as-path access-list 200 permit ^(<our-customer1-AS>_)+$ip as-path access-list 200 permit ^(<our-customer2-AS>_)+$
• beware of the default route!• desi se i najboljima ;-)
• do not accept your own prefixes - they should stay at home• do not accept too specific prefixes• SIX policy: /8 .. /25 for IPv4, /16 .. /48 for IPv6
• block martians• announce your own and nothing else
43sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Register your route objects
44
$ whois -h whois.ripe.net -- '-i or AS2107' | grep ^routeroute: 109.127.192.0/18route: 141.255.192.0/18route: 149.62.64.0/18route: 153.5.0.0/16route: 164.8.0.0/16route: 164.8.0.0/17route: 164.8.128.0/17route: 164.8.128.0/20route: 178.172.0.0/17route: 185.13.52.0/22route: 193.138.1.0/24route: 193.138.2.0/24route: 193.2.0.0/16route: 194.249.0.0/16route: 212.235.128.0/17route: 88.200.0.0/17route: 92.244.64.0/19route: 95.87.128.0/18route6: 2001:1470::/29route6: 2001:1470::/32
44sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Example: What is registered at RIPE?
45
$ peval -h whois.ripe.net -protocol ripe -no-as AS-ARNES((AS28933 AS2121 AS51988 AS42909 AS12785 AS50195 AS2107 ))
$ peval -h whois.ripe.net -protocol ripe 'afi ipv6 AS-ARNES'({2A00:1600::/32, 2A00:1368::/32, 2001:1470::/32, 2001:7F8:46::/48, 2001:67C:64::/48, 2001:678:4::/48, 2001:678:5::/48})
• peval– list of ASNs at the end of the AS-PATH– list of prefixes
45sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Example: What is registered at RIPE?
46
• public whois servers
$ whois -h filtergen.level3.net -- "-v6 RIPE::RS-ARNES-HOSTED"Prefix list for policy RIPE::RS-ARNES-HOSTED = RIPE::RS-ARNES-HOSTED
2001:503:c27::/482001:503:231d::/482001:658:4::/482001:658:5::/482001:67c:44::/482001:7f8:46::/48
46sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Register your peering information
47
47sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Register your peering information
47https://www.peeringdb.com/
47sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
RPKI-based BGP route origin validation
48
48sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
RPKI-based BGP route origin validation
48
https://certification.ripe.net/
48sreda, 04. december 13
Matjaž Straus Istenič, 8.9.2011
Goodies• route server (reflector)• IXP manager• looking-glass router• graphs
– public– or members only– or private
• meetings :-)
49
49sreda, 04. december 13
Presenter Name, Date
Route server• use route server from day 1
• SIX uses bird - http://bird.network.cz• how it works• goodies– enforced policy– community based prefix filtering– “automatic” localization
50
50sreda, 04. december 13
Presenter Name, Date
Bird at SIX
51
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
51sreda, 04. december 13
Presenter Name, Date
Bird at SIX
51
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
All the magic happens here!
51sreda, 04. december 13
Presenter Name, Date
Bird at SIX
51
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
All the magic happens here! All valid routes are here.
51sreda, 04. december 13
Presenter Name, Date
Bird at SIX
51
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
All the magic happens here!
Policy (pipe) filters received routes, marks them, adjusts preference according to location, filters advertised routes.
All valid routes are here.
51sreda, 04. december 13
Presenter Name, Date
Route server• improved (enforced) security– filtering based on routing registry–matching on prefix and origin AS
– blocks martians– blocks default– blocks more specifics
5227
52sreda, 04. december 13
Presenter Name, Date
Route server• improved (enforced) security– filtering based on routing registry–matching on prefix and origin AS
– blocks martians– blocks default– blocks more specifics
5227
It is you who is
responsible for the
security of your
network !
52sreda, 04. december 13
Presenter Name, Date
Example: route server - custom filtering• based on BGP communities
53
description community extended community
Prevent announcement of a prefix to a peer 0:peer-as soo:0:peer-as
Announce a route to a certain peer 51988:peer-as soo:51988:peer-as
Prevent announcement of a prefix to all peers 0:51988 soo:0:51988
53sreda, 04. december 13
Presenter Name, Date
Example: route server - localization• we adjust the route preference according to AS_PATH length
54
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
54sreda, 04. december 13
Presenter Name, Date
Example: route server - localization• we adjust the route preference according to AS_PATH length
54
member 1
R1
R101
member 2
R2
R102
MASTERRIBRS
R101 RTBGP
R2 RT
R102 RT
BGP
BGP
R1 RTBGP
R101 POLICY
R102 POLICY
R2 POLICYR1 POLICY
import from member RT/BGP to master RIB:preference = 100;if bgp_path.len > 50 then preference = 0; else preference = 100 - ( 2 * bgp_path.len );export from master RIB to member RT/BGP:if same_location() then preference = preference + 1;
54sreda, 04. december 13
Presenter Name, Date
IXP Manager• portal and RS manager
55
55sreda, 04. december 13
Presenter Name, Date
IXP Manager• portal and RS manager
55
https://github.com/inex/IXP-Manager/wiki
55sreda, 04. december 13
Presenter Name, Date
Looking glass
56screenshot from NLNOG RING
http://lg.ring.nlnog.net/
56sreda, 04. december 13
Presenter Name, Date
Looking glass
56
https://github.com/sileht/bird-lg/
screenshot from NLNOG RINGhttp://lg.ring.nlnog.net/
56sreda, 04. december 13
Presenter Name, Date
Looking glass• ...or, at least, route-collector
56
https://github.com/sileht/bird-lg/
[email protected]:rc> show route aspath-regex 2107.* active-path terse table inet6.0
inet6.0: 132 destinations, 330 routes (132 active, 0 holddown, 0 hidden)Restart Complete+ = Active Route, - = Last Active, * = Both
A Destination P Prf Metric 1 Metric 2 Next hop AS path* 2001:678:4::/48 B 170 1 0 >2001:7f8:46:0:1::2107 2107 42909 I* 2001:678:5::/48 B 170 1 0 >2001:7f8:46:0:1::2107 2107 42909 I* 2001:1470::/29 B 170 1 0 >2001:7f8:46:0:1::2107 2107 I* 2001:1470::/32 B 170 1 0 >2001:7f8:46:0:1::2107 2107 I* 2a00:1600::/32 B 170 1 0 >2001:7f8:46:0:1::2107 2107 50195 I* 2a00:d440::/29 B 170 1 0 >2001:7f8:46:0:1::2107 2107 58046 I
screenshot from NLNOG RINGhttp://lg.ring.nlnog.net/
56sreda, 04. december 13
Presenter Name, Date
Graphs
57
57sreda, 04. december 13
Presenter Name, Date
Graphs
57
57sreda, 04. december 13
Presenter Name, Date
Graphs
57
57sreda, 04. december 13
Presenter Name, Date
Graphs
57
57sreda, 04. december 13
Presenter Name, Date
Graphs
57
57sreda, 04. december 13
Presenter Name, Date
Graphs
57
Collect data
and graph as
much as you can
57sreda, 04. december 13
Presenter Name, Date
Meet the community
58CC EssjayNZ/flickr
58sreda, 04. december 13
Thank you!
59sreda, 04. december 13