“ocn experience to handle the traffic growth and the future“ · copyright © 2011 ntt...
TRANSCRIPT
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
“ OCN Experience to Handle the Traffic Growth and the Future “
Takeshi Tomochika <takeshi.tomochika(at)ntt.com>Chika Yoshimura <chika.yoshimura(at)ntt.com>
NTT Communications, OCN
23th February 2011
1
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.22
Background
• Internet traffic is growing more and more
• One of the most important missions of ISPs - to carry the traffic with stability & without any
congestion
• Making the backbone robust
• We are talking about:- current traffic situation in Japan- issues at OCN when designing the backbone network- future visions
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN?
3. Current issues we are facing
4. Future visions
5. Wrap up
33
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
879.61234
689.5 860.8319.7 425.4 469.1 523.6 636.6 721.7 812.9 988.4
1453.71362.9
567.5512.5468.7400.5354.3320.2278.8872.4631.5
943.4
0
200
400
600
800
1000
1200
1400
1600
Total Am
ount of Traffic
(G
pbs)
Internet Traffic Trend in Japan
2004 2005 2006 2007 2008 2009 2010
Download
Upload
17.8%
7.5%%%%
4
* Average trafficsource: Internet Traffic Trends in Japan ( MIC* )
MIC: Ministry of Internal Affairs and Communications
• Total amount of Broadband Traffic is 1.46Tbps (Download)- 17.8% growth compared to last year
• Upload traffic decreased over the last half year (872Gbps)
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.55
• Traffic volume per subscriber growing- 16kbps (in 2004) -> 45kbps (in 2010)
Internet Traffic Trend in Japan (cont.)
source: Internet Traffic Trends in Japan ( MIC* )MIC: Ministry of Internal Affairs and Communications
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
05,000,00010,000,00015,000,00020,000,00025,000,00030,000,00035,000,000subscribers
Total 19,557,100 23,301,100 25,744,769 28,303,003 30,115,989 31,709,116 32,040,792 DSL 13,675,800 14,517,800 14,235,925 13,133,113 11,601,734 10,134,491 9,735,140 CATV 2,959,710 3,309,480 3,565,427 3,827,417 4,083,072 4,300,594 4,352,878 FTTH 2,896,930 5,457,690 7,931,837 11,329,886 14,418,215 17,195,696 17,788,535 2004 2005 2006 2007 2008 2009 2010
source: Ministry of information and Communications Statistic Database
FTTH
DSL
Broadband subscribers in Japan
66
• Growing Broadband subscribers• Shifting from DSL (metal) to FTTH (optical fiber)
Internet Traffic Trend in Japan (cont.)
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Overview
• Internet traffic in Japan has been growing consistently
• Traffic will keep rising in the future- ISPs have to …
• design a robust backbone network to deal with the situation
• How backbone we have been making?• How bandwidth we have?
7
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN?
3. Current issues we are facing
4. Future visions
5. Wrap up
88
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
67%54%100% 100% 100%
NTT EAST NTT WEST NTT DATA
NTT Holding
: Domestic Open IP Service Brand
NTT Communications
-Domestic Long
Distance
-International
-Internet
-Domestic Local
(Eastern Japan)
-Domestic Local
(Western Japan)
-Systems
Integration
-Mobile
: International Open IP Service Brand
Outline of NTT Group
http://www.hk.ntt.com/en/index.html100%
NTT docomo
100%http://www.hknet.com/en/home/index.htm
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
HoustonHouston
AustraliaAustralia
TaiwanTaiwan
KoreaKorea
IndonesiaIndonesia
DusseldorfDusseldorf
MadridMadrid
US BackboneUS Backbone
EUROPE BackboneEUROPE Backbone
EquinixEquinixEquinixEquinix San JoseSan JoseSan JoseSan JoseNASANASANASANASAPAIXPAIXPAIXPAIX
EquinixEquinixEquinixEquinix ChicagoChicagoChicagoChicago EquinixEquinixEquinixEquinix AshburnAshburnAshburnAshburnEquinixEquinixEquinixEquinix DallasDallasDallasDallas
DEDEDEDE----CIXCIXCIXCIXAMSIXAMSIXAMSIXAMSIX PARIXPARIXPARIXPARIXESPANIXESPANIXESPANIXESPANIXLINXLINXLINXLINXPhilippines
Philippines
MalaysiaMalaysia
VietnamVietnam
IndiaIndia
ThailandThailand
GenevaGeneva
ParisParis
FrankfurtFrankfurt
SingaporeSingapore
EquinixEquinixEquinixEquinix New YorkNew YorkNew YorkNew YorkLondon
LondonHK IXHK IXHK IXHK IXAtlantaAtlanta
AmsterdamAmsterdam
OCN愛知OCN Aichi
OCN愛知OCN Aichi
OCN KyotoOCN Kyoto
OCN OsakaOCN Osaka
OCN HyogoOCN Hyogo
OCN HiroshimaOCN Hiroshima
OCN FukuokaOCN Fukuoka
OCN HokkaidoOCN Hokkaido
OCN KanagawaOCN Kanagawa
OCN MiyagiOCN Miyagi
OCN ChibaOCN Chiba
OCN TokyoOCN Tokyo
OCN Tokyo westOCN Tokyo west160Gbps
160Gbps
20Gbps
20Gbps
20Gbps
20Gbps
400Gbps
400Gbps
20Gbps
20Gbps
60Gbps
60Gbps
120Gbps
120Gbps
60Gbps
60Gbps
1Gbps
NSPIXP-3
40Gbps
JPNAP Osaka
130Gbps
JPNAP
10Gbps
dix-ie
Osaka GWOsaka GW
ICP
20Gbps
20Gbps
80Gbps
80Gbps
440Gbps
440Gbps
20Gbps
20Gbps
ChinaChina
Hong KongHong Kong
San JoseSan Jose
SeattleSeattle
ChicagoChicago
New YorkNew York
Washington D.C.Washington D.C.
DallasDallas
MiamiMiami
Los AngelesLos Angeles
OsakaOsaka
Osaka CoreOsaka Core
Aichi CoreAichi Core
Gunma CoreGunma Core
Tokyo CoreTokyo Core
Tokyo GWTokyo GW
ASIA BackboneASIA Backbone
TokyoTokyo
国内接続国内接続国内接続国内接続Domestic External 1Gbps
Internet Multifeed
NTT Communications’ IP Backbone Network
November 2010
340Gbps
340Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
380Gbps
380Gbps
170Gbps
220Gbps
OCN Backbone
ntt.net
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
0
100
200
300
400
500
600
700
800
Oct-97 Oct-98 Jul-99 Jan-00 Jul-00 Jan-01 Jul-01 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Jul-04 Jan-05 Jul-05 Jan-06 Jul-06 Dec-06 Aug-07 Dec-07 Jun-08 Nov-08 May-09 Nov-09 May-10 Nov-10
Gbps
between east and west
ntt.net
IX
Bandwidth history of OCN
11
Jul 2001
Started OCN IPv6
Tunnel Commercial
Service
Mar 2003
installed 10G-IF in the backbone
Feb 2010
300G between Japan to the U.S.
(ntt.net)
Jan 2011
400G between
Japan to the U.S.
(ntt.net)
Feb 2007
100G between
Japan to the U.S.
(ntt.net)
Nov 2001
Started
FTTH service
Jan 2001
Started OCN ADSL
Service
Dec 1999Started OCN IPv6
Tunnel Trial Service
Dec 1996
Started OCN ×2400×2400
×5800×5800
×14000×14000
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Make our network larger and larger as Internet traffic grows
• Issues we have been facing
• Efforts we have been making
12
What we are doing
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN?
3. Current issues we are facing
4. Future visions
5. Wrap up
1313
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
1. Scalability of Router Forwarding Tables
2. Link Aggregation
14
The issues we are facing
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
FIB table of OCN
• FIB(Forwarding Information Base) table has been growing
• Causes of growing FIB1. BGP full routes (more than 340,000 in February 2011)
2. Prefixes with no-export
3. ECMP, {i, e} bgp-multipath
source: http://bgp.potaroo.net/
Growth of the BGP Table - 1994 to Present
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Scalability of Router Forwarding Tables
• When a rerouting event occurs, potentially thousands of routes must be updated
• It took a lot of time to converge the routes
FIB of router-A
prefix output interface(s)
10.1.0.0/16
IF-1
IF-2
LAG-3(IF-4, 5)
10.2.0.0/16
IF-1
IF-2
LAG-3(IF-4, 5)
IF-1
10.1/1610.2/16…
LAG-3(IF-4,5)
IF-2
router A
router B router Crouter D
××××
××××
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• We were facing a problem:- OSPF neighbor went down due to FIB table convergence
• Between router A and B- Link Aggregation (LAG) had been enabled (minimum-links = 1) - OSPF neighbor had been connected through the LAG interface
• When all member-links but one had been to make disabled- We had expected the OSPF neighbor to remain up
17
routerrouterrouterrouter
AAAA
routerrouterrouterrouter
BBBB
membermembermembermember----link 1link 1link 1link 1
Link AggregationLink AggregationLink AggregationLink Aggregation
membermembermembermember----link 2(down)link 2(down)link 2(down)link 2(down)
membermembermembermember----link 3(down)link 3(down)link 3(down)link 3(down)
membermembermembermember----link 4 (down)link 4 (down)link 4 (down)link 4 (down)
membermembermembermember----link 5 (down)link 5 (down)link 5 (down)link 5 (down)
membermembermembermember----link 6 (down)link 6 (down)link 6 (down)link 6 (down)
Scalability of Router Forwarding Tables
OSPF neighbor
went down
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
18
• What happened?
routerrouterrouterrouter
AAAA
routerrouterrouterrouter
BBBB
membermembermembermember----link 1link 1link 1link 1
Link Aggregation InterfaceLink Aggregation InterfaceLink Aggregation InterfaceLink Aggregation Interface
membermembermembermember----link 2link 2link 2link 2
membermembermembermember----link 3link 3link 3link 3
membermembermembermember----link 4link 4link 4link 4
membermembermembermember----link 5link 5link 5link 5
membermembermembermember----link 6link 6link 6link 6
disable
(1) Router A detected the
interfaces were down
(2) Router A started updating FIB
(3) Router A finished
updating FIB
(4) Router A
chose another interface to send
OSPF hello
OSPF hellomore than OSPF dead-timer
Router-A could not send any OSPF hello packets during (1) – (3), then the neighbor went down
Scalability of Router Forwarding Tables
OSPF hello
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Hierarchical FIB - Cisco: BGP Prefix Independent Convergence(PIC)- Juniper: indirect-nexthop
For more information: BGP Convergence in much less than a secondhttp://www.nanog.org/meetings/nanog40/presentations/ClarenceFilsfils-BGP.pdf
• Fewer routes to be updated
• Improving the route convergence time
Scalability of Router Forwarding Tables
19
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Link Aggregation Issues
20
• A lot of Link Aggregation 10GE Interfaces in the backbone
OCN愛知OCN Aichi
OCN愛知OCN Aichi
OCN KyotoOCN Kyoto
OCN OsakaOCN Osaka
OCN HyogoOCN Hyogo
OCN HiroshimaOCN Hiroshima
OCN FukuokaOCN Fukuoka
OCN HokkaidoOCN Hokkaido
OCN KanagawaOCN Kanagawa
OCN MiyagiOCN Miyagi
OCN ChibaOCN Chiba
OCN TokyoOCN Tokyo
OCN Tokyo westOCN Tokyo west160Gbps
160Gbps
20Gbps
20Gbps
20Gbps
20Gbps
400Gbps
400Gbps
20Gbps
20Gbps
60Gbps
60Gbps
120Gbps
120Gbps
60Gbps
60Gbps
1Gbps
NSPIXP-3
40Gbps
JPNAP Osaka
130Gbps
JPNAP
10Gbps
dix-ie
Osaka GWOsaka GW
ICP
20Gbps
20Gbps
80Gbps
80Gbps
440Gbps
440Gbps
20Gbps
20Gbps
Osaka CoreOsaka Core
Aichi CoreAichi Core
Gunma CoreGunma Core
Tokyo CoreTokyo Core
Tokyo GWTokyo GW
OCN Backbone
国内接続国内接続国内接続国内接続Domestic External 1Gbps
Internet Multifeed
340Gbps
340Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
200Gbps
380Gbps
380Gbps
• Traffic balance issues (Traffic Polarization)
• Operation issues
• Other issues
20
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 21
LAG Issues
~traffic balance issues (1/3)~~~~
• Traffic balance in the LAG(1)
– Can’t use per-packet round-robin
• Simple round-robin bring about packet reordering in a flow
– Hashing algorithm: calculate the hash value based on the packet information (IP
address, MAC address, and etc.) to decide Output I/F
• Traffic are distributed per flow using the hash values
� Issue 1: traffic-unbalance by variation of flow
4 links
Packet
IP MACPayload
to LAG-A calcurate thehash
hash=1 → Out I/F=e1
hash=2 → Out I/F=e2
・・・・・・
LAG-A
Flow
Packet
LAG
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.22
• Traffic balance in the LAG(2)
– Issue 2: The less # of hash elements, the worse traffic-balanced
�as a result, less effective use of bandwidth
5 links LAG 4 links LAG 3 links LAG
IF#1 H1、H6
IF#2 H2、H7
IF#3 H3、H8
IF#4 H4
IF#5 H5
IF#1 H1、H5
IF#2 H2、H6
IF#3 H3、H7
IF#4 H4、H8
IF#1 H1、H4、H7
IF#2 H2、H5、H8
IF#3 H3、H6
2:2:2:1:1
10+10+10+10*1/2+1
0*1/2=40
2:2:2:2
10+10+10+10=40
3:3:2
10+10+10*2/3=26.7
Traffic cannot be evenly
distributed due to the
hash mechanism
e.g.: Traffic balance in a LAG when # of hash elements is 8
Only use 40G / 50G
←Traffic balance ratio←Effective bandwidth in the LAG
Only use 27G / 30G
LAG Issues
~traffic balance issues (2/3)~~~~
22
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.23
cf. Difference in traffic balance by # of hash elements
5 links LAG 4 links LAG 3 links LAG
IF#1 H1,H6
IF#2 H2,H7
IF#3 H3,H8
IF#4 H4
IF#5 H5
IF#1 H1,H5
IF#2 H2,H6
IF#3 H3,H7
IF#4 H4,H8
IF#1 H1,H4,H7
IF#2 H2,H5,H8
IF#3 H3,H6
40 40 26.7
5 links LAG 4 links LAG 3 links LAG
IF#1 H1,H6,・・・H26,H31
IF#2 H2,H7,・・・H27,H32
IF#3 H3,H8,・・・H28
IF#4 H4,H9,・・・H29
IF#5 H5,H10,・・・H30
IF#1 H1,H5,・・・H29
IF#2 H2,H6,・・・H30
IF#3 H3,H7,・・・H31
IF#4 H4,H8,・・・H32
IF#1 H1,H4,・・・H28,H31
IF#2 H2,H5,・・・H29,H32
IF#3 H3,H6,・・・H30
7:7:6:6:6
10+10+10*6/7+10*6/7+
10*6/7=45.7
8:8:8:8
10+10+10+10=40
11:11:10
10+10+10*10/11=29.1
A large number of hash elements is better
LAG Issues
~traffic balance issues (2/3)~~~~
←Traffic balance ratio←Effective bandwidth
in the LAG
e.g.1: Traffic balance in a LAG when # of hash elements is 8
e.g.2: Traffic balance in a LAG when # of hash elements is 32
23
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.24
• Traffic balance by ECMP (Equal Cost Multi Path) and LAG: Case1
– If calculation logic of LAG is the same as ECMP’s, it will bring about unbalanced traffic in physical links
LAG1
LAG2
calc. for ECMP①
calc. for LAG①
unbalanced
(in LAG)
evenly balance(ECMP)
same logic
Some routers have the same calculation
logics for ECMP and LAG as a default
LAG Issues
~~~~traffic balance issues (3/3 - 1)~~~~
flow 1flow 2
flow 3flow4
flow 1flow 2
flow 3flow 4
24
no traffic
no traffic
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Traffic balance by ECMP and LAG : Case2– If calculation logic of ECMP is the same as that of previous ECMP, it will
bring about unbalanced traffic
25
LAG
LAG
LAG Issues
~~~~traffic balance issues (3/3 - 2)~~~~
calc. for ECMP①
calc. for ECMP①
calc. for LAG②
****change****
same logic
unbalanced(ECMP)
flow 1
flow 2
25
no traffic
no traffic
flow 1flow 2
flow 3flow 4
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Traffic balance by ECMP and LAG : Case3– If calculation logic of LAG is the same as that of ECMP at the previous node, it
will bring about unbalanced traffic
26
LAG
LAG
calc. for ECMP①
calc. for ECMP③
same logic
calc. for LAG②
unbalanced(in LAG)
calc. for LAG①
****change****
Need to consider balance logics, network topology, configurations
LAG Issues
~~~~traffic balance issues (3/3 - 3)~~~~
flow 1
flow 2
26
unbalanced(in LAG)
flow 5
no traffic
no traffic
flow 1
flow 2
flow 3flow 4
flow 5
* Some latest routers can include a router-ID in the seed of hash to avoid case 2,3
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.27
• LAG operation (1)
– In the case of silent-failure, traffic through the fault link will drop
– LACP (Link Aggregation Control Protocol)
・ Sending and receiving control
frames in physical links
・Attention to Interoperability
– BFD Per Member Link
(Bidirectional Forwarding Detection)
transmission device
LAG Issues
~~~~operation issues (1/3)~~~~
RouterRouter LAG-I/F
27
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.28
• LAG operation (2)
– Switching policy of LAG-I/F
• minimum-link (trunk-threshold)
• threshold whether LAG-I/F is up or down
• This switching policy is important for
effective use of LAG
• consider the entire network topology
e.g.: minimum-link when the policy is 70% in LAG
(1)Normally, packets are forwarded to all the link-up I/Fs
(3) LAG I/F goes down, and
traffic move
minimum-link = 3
LAG Issues
~~~~operation issues (2/3)~~~~
28
# of links in LAG 3 4 5 6 7 8 9 10
minimum-link 3 3 4 5 5 6 7 7
LAGLAG
(2) still LAG is up, as # of up-links is not less than 3
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.29
• LAG operation (3)
– Ping for test
• Packet goes through only one physical
interface
• Need to test each interface with letting the
rest go down
• expect Ethernet OAM
LAG Issues
~~~~operation issues (3/3)~~~~
29
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.30
• Limitations on # of links in a LAG
• Issues of physical wiring
– Increased # of physical links
-> Complicated maintenance
• Need a well-thought-out plan for LAG
– How to assign physical links to Line Cards
• based on redundant policy
• MTBF for each part
• Cost
– e.g. Policy 1: keep LAG-I/F up as much as possible
• assign each physical link to each LC, minimum-link = 1
– e.g. Policy 2: Switching traffic to the other LAG immediately
• assign all physical links to one LC, minimum-link = # of links
– e.g. Policy 3: Between policy 1 and policy 2
• LAG is troublesome
NOTE: this is NOT
NTT Communications’
equipment
LAG Issues
~~~~other issues~~~~
30
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN?
3. Current issues we are facing
4. Future visions
5. Wrap up
3131
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.32
How to handle traffic growth
32
(1) Change Network • new routers, new switches
• new router-interface
(100GE)
(2) Control Traffic • Cache Servers
• CDN
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.33
Expectation for 100GE
• Need 100GE I/F– Bandwidth over 1Tbps
– LAG is troublesome
• Request
– Lower price
• CFP is expensive
• LR10
– Support long-distance transmission (ER4)
– Higher Capacity
• Capacity per chassis will be decreased when migrating from 10GEs to 100GEs in some current routers
– LAG of 10GE and 100GE simultaneously
– Interoperability, 100GE LAG, Ether OAM
– Next step: 400GE, 1T Ether
33
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
The Best award of The Best award of InteropInterop Tokyo for 2 years in a rowTokyo for 2 years in a row
2009 100GE-SR10 demonstrated with transmission equipmentand traffic generator
2010 100GE transmission network (100GE-LR4) was providedfor practical operation
2010/6/8 News Release
NTT Com, Infinera and Ixia to Provide World’s First Practical
100 Gbps Ethernet Interconnection at Interop Tokyo 2010
Our preparation for 100GE
34
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
transitpeer
users
(1) transit/peer sides
(2) backbone
(3) edge sides
Cache servers
• Legal changes in January 2010 in Japan - became legal to install cache servers
without permission(s)
• Demerits not to install cache servers- Streaming delays because of carrying the
traffic from peer/transit network- more bandwidth- costs for transit network
• Merits to install cache servers- Transit cost saving,- Bandwidth saving- Fixing delay
• Issues1. Equipment performance (cache hit ratio,
lack of bandwidth… )2. Where to place3. When equipment failure
35
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN?
3. Current issues we are facing
4. Future visions
5. Wrap up
3636
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Wrap up
• The total traffic in Japan has been consistently increasing.- The traffic will keep growing in the future.
• We are continuing to design a strong backbone network.- But we have some designing/operational issues
• We are going to need 100GE in the near future to deal with the situation.
• How is your network? Do you have any ideas or suggestions to cope with the expected growth of traffic in the future?
37
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.38
Thank you!Thank you!Thank you!Thank you!