internet exchanges: how hard can it be to build a resilient infrastructure in a small country ?
DESCRIPTION
Riga , Latvia April 19 , 2013. Internet Exchanges: how hard can it be to build a resilient infrastructure in a small country ?. Uģis Bērziņš, CCIE #8972 Emeritus Chairman of The Board Santa Monica Networks. IXP History in Latvia. Internet Exchanges in Latvia. - PowerPoint PPT PresentationTRANSCRIPT
1
Riga, LatviaApril 19, 2013
Uģis Bērziņš, CCIE #8972 EmeritusChairman of The BoardSanta Monica Networks
Internet Exchanges:how hard can it be to build a resilient infrastructure ina small country?
IXP History in Latvia
Internet Exchanges in Latvia• GIX – LATNET / LU MII, Raiņa blvd. 29 (1992/1993)
Natural peering with AS2588
• RIX – BKC, Pērses 2 (1997)Separate L2 switch, 100 Mbps Fiber, no Route server
• LIX (v1) – cooperation of 4 ISPs: Latnet, Lattelecom, Microlink, Telia Latvia (2003)Four L2 switches, 1 Gbps, no dedicated Route Server
• SMILE1 – Santa Monica Networks – non-ISP ownership (2005)Two L3 switches, Nx1Gbps, AS39626, Switch = Route Server
• LIX (v2) – cooperation of 3 ISPs: Latnet, Lattelecom, Telia Latvia (2008?)Three L3 switches, the same design as SMILE1, 10G Core
• SMILE2 – Santa Monica Networks (2010-2013)Two L2 switches, AS39626, Two Route Server, Design similar to DE-CIX
From simple things To complex… and back
LIX (v1)Lessons Learned
LIX (v1) Topology
2
1
4
3
6
5
8
7
10
9
UTILSTAT
DUPLXSPEED
SYSTEMRPS
MODE 11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTILSTAT
DUPLXSPEED
SYSTEMRPS
MODE 11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTILSTAT
DUPLXSPEED
SYSTEMRPS
MODE 11 12
CATALYST 3550
2
1
4
3
6
5
8
7
10
9
UTILSTAT
DUPLXSPEED
SYSTEMRPS
MODE 11 12
CATALYST 3550
GEGE
GE
X X
Layer 2STPBPDU Filter on edge
LIX (v1) Developments
During lifetime there was increase of Core link bandwidth by using link aggregation
Layer 2 loop prevention – STP did not help
No Route Server(s) – any to any eBGP
No single point of contact
Restriction based on polytical rather than technical conditions
Platform growth potential limited
Frequent service stability issues with Link Aggregation and L2 Loops
Issues
SMILE1Lessons Learned
SMILE1 Topology
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT
SUPERVISOR 2
WS-X6 K-S2-MSFC2 /2
STAT
US
SYST
EM
ACT
IVE
PWR
MG
MT
RES
ET
CONSOLE
PORT 1 PORT 2
PCMCIA EJE CTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
SUPERV ISOR 2
WS -X6 K-S2 -M SFC2
STA
TUS
SYS
TEM
ACT
IVE
PWR
MG
MT
RES
ET
CONSOLE
PORT 1 PORT 2
PCMCIA EJE CTMODEPORT
CONSO LE
SWITCH LOAD
1%
100%
STATUS
WS-X6 41 6-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6 41 6-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
INPUT
SUPERVI SOR 2
WS-X6K-S2 -MSFC2/2
STAT
US
SYS
TEM
ACTI
VE
PWR
MG
MT
RES
ET
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMO DEPORT
CONSOLE
SWITCH LOAD
1%
100%
SUPERVISOR 2
WS-X6K-S2-MSFC2
STA
TUS
SYS
TEM
ACTI
VE
PWR
MG
MT
RES
ET
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
STATUS
WS-X6 41 6-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LI NK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6 41 6-GBIC
16 PORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LI NK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
Layer 3No STP neededSwitch = Route Server
SMILE1 Developments
Targets set:
• High Availability and Flexibility
• No political restrictions
• Better use of line capacity
Totally different approach than LIX
Stability of service was superior – one unplanned downtime of a single node during major power supply disaster
Use of BGP Communities
Performance – Platform limited to Nx1G
Routing Flexibility:
• eBGP Peering directly with SMILE switch
• SMILE switch selects best path – limited possibilties by SMILE customers to influence routing decision
Issues
LIX (v2) response
LIX (v2) Topology
Layer 3No STP neededSwitch = Route Server
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTA
LL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT
SUP ERVIS OR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONS OLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUPERV ISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONS OLE
SWITCH LOAD
1%
100%
LINK
LINK
STATUS
WS-X6416-GB IC
16 P ORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GB IC
16 P ORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTA
LL
INPUT
SUPERVISOR 2
WS-X6K-S 2-MSFC2/2
ST
AT
US
SY
STE
M
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUP ERVIS OR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
STE
M
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONS OLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
STA TUS
WS-X6416-GBIC
16 PORT GIGA BIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STA TUS
WS-X6416-GBIC
16 PORT GIGA BIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
1
2
3
4
5
6
FANSTATUS
Power Supply 1 Power Supply 2
Catalyst 6500 SERIES
100-240 V~16 A
60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT100-240 V~
16 A60/50 Hz
INPUTOK
FANOK
OUTPUTFAIL
RUNINSTALL
INPUT
SUP ERVIS OR 2
WS-X6K-S2-MSFC2/2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
SUPERV ISOR 2
WS-X6K-S2-MSFC2
ST
AT
US
SY
ST
EM
AC
TIV
E
PW
R M
GM
T
RE
SE
T
CONSOLE
PORT 1 PORT 2
PCMCIA EJECTMODEPORT
CONSOLE
SWITCH LOAD
1%
100%
LINK
LINK
STATUS
WS-X6416-GBIC
16 P ORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
STATUS
WS-X6416-GBIC
16 P ORT GIGABIT ETHERNETLINK LINK
1
2
1 2
LINK LINK
15
16
15 16
LINK LINK
13
14
13 14
LINK LINK
11
12
11 12
LINK LINK
9
10
9 10
LINK LINK
7
8
7 8
LINK LINK
5
6
5 6
LINK LINK
3
4
3 4
SMILE2
SMILE2 Topology
MED transparencyAS Path transparency
SMILE2 Developments
Targets set:
• Go for 10G
• High Availability and Flexibility
• No political restrictions
• Better use of line capacity
Return to well proven DE-CIX model (L2)
Route Servers, BGP Communities
Customers fully control routing decision
Direct peerings between major players
Major players acquire small ISPs
Going commercial
Issues
Tools can be different
Thank you.