ipodwdm and ip ngn core design - cisco.com · the quality of ip ngn design •hierarchy (p is...

Cisco Confidential 1© 2010 Cisco and/or its affiliates. All rights reserved.

IPoDWDM and IP NGN Core designJosef UngermanCisco, CCIE #6167

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

• Motivations for IP NGN• Trends is IP/MPLS Core Design• Making Routers Cheaper• IPoDWDM • Pre-FEC proactive protection• 100GE and 40GE• Future


IP Traffic will increase 4x from 2009 to 2014(34% CAGR)

Mobile Data Traffic will double every

year

IP Traffic grew 45% in 2009

Video will be 66% of mobile data, and 91% of global traffic

Central & Eastern Europe will be

2.3EB/month (3.6% of the global 64EB/month)Source: Cisco Visual Networking Index—Forecast, 2009–2014 [Cisco VNE]

IP Traffic

Internet TrafficBusiness IP traffic will increase 2.6x (21% CAGR), only video will grow 10x


1) Reduce the number of networks� the largest portion of the traffic will rule the design

2) Reduce the number of layers� the largest portion of the traffic must be as close to fiber as possible, eliminate overlay

3) Reduce the number of nodes� the largest portion of the traffic must pass the lowest possible number of nodes

4) Reduce the number of links� the number of changes in the network must be minimal when adding more capacity; use statistical multiplex - small traffic follows big traffic

5) Use modern technologies� Watch TCO, Price/Performance ratio, Watt/Gigabit ratio, investment protection (h/w upgradeability, s/w roadmaps)

How to move bits cheaper......reduce OPEX, CAPEX, and keep reasonable quality?


• IP NGN = Single Multiservice NetworkNot multiple single-service networks. Key enablers are Virtualization, QoS and Security.

• IP NGN = MPLS + DWDM Optical TransportMoving bits cheaper. 100GE evolution, Price/Gigabit and Watt/Gigabit reduction.

• IP NGN = optimal balance between Clouds and CircuitsStatistical Multiplex (IP/MPLS, MPLS-TP) and Division Multiplex (DWDM, OTN).


Designing End to End IP/MPLS networks

Hierarchical Design (divide & conquer approach)

PE is always connected to P

Simple upgrades

Static Transport LayerProtection is in the IP domain

Hierarchy Hollow Core ideaRouter Bypass

• Hierarchical Design with optical router bypass

• Still simple upgrades• Cheaper bandwidth• Quality is kept

Flat Design (full mesh between IP routers)

Fewer nodes, Much more Links

Complex upgrades, complex QoS

Dynamic Transport Layer (G.MPLS, VCAT/ODU-FLEX)

Protection in the Optical domain


MetroAggregation

Core

BNG(Edge)

Internet Gateways

The Quality of IP NGN Design• Hierarchy (P is connected to PE)• One network, one IGP (not multiple)• QoS and Security everywhere• Scalability – where will 40/100GE start?

The Quality of IP NGN Design• Hierarchy (P is connected to PE)• One network, one IGP (not multiple)• QoS and Security everywhere• Scalability – where will 40/100GE start?

SSN – Single Service Node (eg. BRAS, IGW)MSN – Multiservice Node (eg. P router)•SSN-MSN link = ok safe operation

•MSN-MSN link = think twice!such link needs proper QoS and capacity mgt

•SSN-SSN link = stop! don’t break the hierarchy, don’t create another net


Creates an IP mesh

Creates an IP hierarchy

Statistical Multiplexing

DWDM or dark fiber – direct link(optical layer bypass)

Too large distances – OTN switch(transport layer bypass)


100GE� IEEE 802.3ba ratified � CRS-3 today, in 2011/12 also ASR9K & Nexus7K

OTN (Optical Transport Network)� OTN Framing

� implemented today for Ethernet interfaces – ITU-T G.709 � CRS, ASR9000, 7600, CPT, ONS (ODU2e, ODU3, future ODU4)

� OTN Aggregation� implemented today for 10GE � ODU3 (future Any-Rate, ODU4, MPLS-TP)� ONS15454 Muxponder

� OTN Switching� geographically very large countries, or very dense 10G E-Line networks (G.MPLS)� developed for next generation portfolio


• All lambdas upgraded to 100Gbps• Sub-100G services provided by OTN OEO

AdvantagesAll lambdas on a fibre are 100G

Disadvantages100TXP investment upfrontNeed an additional OTN OEOAll 10G TXPs are obsolete

10G and 100G DWDM Coexistence

� 10G and 100G lambdas co-exist on same fibre� Packet uses 100G, everything else 10G

AdvantagesOnly high demand clients upgraded to 100GProtects existing 10G DWDM investmentLowest cost per bit (100G TXPs>10 x10G TXPs)

DisadvantagesNeed a guard band between 10G and 100G

frequenciesNot appealing in ULH environments

100G lambda

10G lambda

100G lambdas

OTN OTN

10G SR

100G SR

OTN Multiplexing


��

�

�

NoNo

NoNo

No

� No


35bn

30bn

25bn

20bn

15bn

10bn

5bn

� Wholesale and retail Ethernet services : E-Line, E-Tree and E-LAN

� ~90% of Ethernet market place <1GE in 2013� What is the most efficient way to support these

Ethernet Services? OTN circuits or Packets?Source Infonetics 2009 and Cisco VNI

domain for OTN

domain for MPLS

Circuit

Packet

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 14

Routers


• Silicon has fundamentally followed Moore’s law• Optics is fundamentally an analog problem

Routers: 23% Cumulative Average $/Gbps Drop per year / fewer ASICsOptics: $/G stays flat (best case) or increases from one technology to the next

Cisco Core Router Example

10G/40G/100G Networking Ports Biannual Worldwide and Regional Market Size and ForecastsMay 2010

NEW


1) Compact Anatomy� RSP, Route/Switch Processor (instead of RP and FC)� Ethernet-oriented Linecard (non-modular, less memory)

2) Linecard Architecture� 4x 10G NPU (instead of 1x 40G NPU)� one full-duplex NPU shared for rx and tx (instead of 2 dedicated NPU’s)� 2x 40G fabric interface (instead of 1x 80G fabric interface)

3) Special Core-facing Linecards� 8/16 queues per port (instead of thousands)� lower-scale NPU (no need for thousands of interfaces)� licenses for features that not everybody uses (IPoDWDM, SyncE, VPN,...)

4) Oversubscribed Cards� 2:1 ingress overbooking (eg. GPON Aggregation or Intra PoP PE links)

5) Power Consumption� newer chips, fewer chips = 50+% Watt-per-Gigabit savings

© 2010 Cisco Systems, Inc. All rights reserved.BRKIPM-2012 17

Cisco CRSvery modular router anatomy

IOS IOS IOS IOS

RP (active) RP (standby)

MSC-140 –140G

FP-140– 140G

SPA

SPA

SIP-800

midpla

ne 40

G

MSC-40

NP

NP

Qbuff. IOS

Q

Qbuff.

FP-40

NP

NP

Qbuff.

IOS

Q

Qbuff.

NP’

NP’

Qbuff. IOS

Q

Qbuff.

NP’

NP’

Qbuff.

IOS

Q

Qbuff.

PLIM

midpla

ne 40

G

14x 10GE

Switch Fabric Cards(all 8 active)

4, 8 or 16 Linecard slots + 2 RP slots

100GE

QFA (Quantum Flow Array)- 140 Gbps, 125 Mpps NPU- one for RX, one for TX

processing

QFA (Quantum Flow Array)- 140 Gbps, 125 Mpps NPU- one for RX, one for TX

processing

141G rx225G tx141G rx225G tx

midpla

ne 14

0G

midpla

ne 14

0G

© 2010 Cisco Systems, Inc. All rights reserved.BRKIPM-2012 18

Cisco ASR9000compact router/switch anatomy

IOS IOS

RSP (active)

IOS IOS

RSP (fab. active)

NP

NP

NP

NP

buff.

buff.

buff.

buff.

IOS

Transport LC – 40G

NP

NP

NP

NP

buff.

buff.

buff.

buff.

IOS

NP

NP

NP

NP

buff.

buff.

buff.

buff.

Transport LC – 80G

4 or 8 Linecard slots

90G90G

Trident NPU- 15 Gbps, 14 Mpps (2x)- shared for RX & TX processing- more independent NPU’s per card

Trident NPU- 15 Gbps, 14 Mpps (2x)- shared for RX & TX processing- more independent NPU’s per card

NP

NP

buff.

buff.

Transport LC – 16x TGE OSNP

NP

buff.

buff.

IOS NP

NP

buff.

buff.

NP

NP

buff.

buff.

NP

NP

NP

NP

buff.

buff.

buff.

buff.

IOS

Transport LC – 8x TGE OS

RSP (Route/Switch Processor)• CPU + Switch Fabric• active/active SF

RSP (Route/Switch Processor)• CPU + Switch Fabric• active/active SF


Edge-facing Card Core-facing Card Over-subscribed CardCRS-1 EMSE MSC40 $50K FP40 + 4xTGE $19K FP40 + 8xTGE $16K7600 ES+ 4TG $39K ES+T 4TG $16K - -CRS-3 EMSE MSC140 $29K FP140 + 14xTGE $18K FP140 + 20xTGE $15KASR9000 A9K-8T-B $16K A9K-8T-L $9.25K A9K-8T/4-L $5.75K

Watt per TGE (max.)CRS-1 MSC40 + 4xTGE 125 WCRS-1 FP40 + 4xTGE 105 W7600 76-ES+4T 100 WASR9000 A9K-8T 78 WCRS-3 MSC/FP + 14xTGE 43 W


IPoDWDM


Invest in High Capacity SDH/SONET/OTN10 transponders needed4-14 Short Reach opticsEvery Lambda OEOAddt’l transponder & SR for each λExpensive switch w/active electronics

OTN OEO SDH/SONET Solution

Short ReachOptics I/F

Cross Connect(XC)

Transpondersor DWDM I/F

Router

Continue to Invest in XCs & Transponders

Invest in IPoDWDM0 transponders needed2 Tunable DWDM interfaces in routerAll pass-through traffic stays opticalROADM full provisioned, no truck rollsExpensive switch eliminated

IPoDWDM Solution

ROADMs

TunableDWDM I/F

Router

Eliminate Unnecessary OEO XC & Transponders


CoreRouter

Electrical XC

Metro Network

IP Layer ManagementIP Layer Management

P2P DWDM

Optical Layer ManagementTransponders converting short reach to λ

Electrical switching – OEO conversions

Metro Network

Manual patching of 10G connections


Metro Network

Metro Network

Integrated transponders

Photonic switching –no OEO conversions

ROADM

CoreRouter

Common Network Management and Control

MeshROADM


• Integration of core routers with optical transport platform� 2 layer in one � reduced OPEX� Eliminates need O-E-O modules (transponders) in transport platform� Integration at control plane level (pre-FEC FRR) to improve network resiliency

• Increased rack space and power efficiency• Possible integration with 3rd party transport equipment• Improves OSNR, CD and PMD robustness through use of advanced

modulations for high speed channels� ODB, DPSK+ for 40 Gbps� CP-DQPSK planned for 100 Gbps

• Available for CRS, ASR 9000, 7600, and 12000


BeforeBefore

RouterRouter ROADMROADMTransponderTransponder

Transponder Integrated into Router

Transponder Integrated into Router

RouterRouter ROADMROADM

DWDM

I/F


Ensure you are user of user group in proper task groupConfigure DWDM controller using CLI:

controller dwdm0/15/0/0admin-state in-servicewavelength 7

Configure L3 interface same as before


ITU-T G.709Performance Monitoring

Line Card State

show controllers dwdm 0/15/0/0 Port dwdm0/15/0/0Controller State: upLoopback: None

G709 Status OTU

LOS = 0 LOF = 0 LOM = 0 BDI = 0 IAE = 0 BIP = 0 BEI = 0 TIM = 0

ODUAIS = 0 BDI = 0 OCI = 0LCK = 0 BIP = 0 BEI = 0 PTIM = 0 TIM = 0

FEC Mode: Enhanced FEC(default)EC(current second) = 4063 EC = 74084864668 UC = 0 pre-FEC BER = 9.53E-8 Q = 5.26 Q Margin = 5.49

Remote FEC Mode: UnknownFECMISMATCH = 0


Optical Alarms Trace & Performance Monitoring

On-Board TDC included on 2nd Generation

show controllers dwdm 0/15/0/0 (cont)Port dwdm0/15/0/0

Detected Alarms: NoneAsserted Alarms: None

Alarm Reporting Enabled for: LOS LOF LOM IAE OTU-BDI OTU-TIM OTU_SF_BER OTU_SD_BER ODU-AIS ODU-BDI OCI LCK PTIM ODU-TIM FECMISMATCH BER Thresholds: OTU-SF = 10e-3 OTU-SD = 10e-6

OTU TTI Sent String ASCII: Tx TTI Not Configured OTU TTI Received String ASCII: Rx TTI Not Recieved OTU TTI Expected String ASCII: Exp TTI Not Configured

ODU TTI Sent String ASCII: Tx TTI Not Configured ODU TTI Received String ASCII: Rx TTI Not Recieved ODU TTI Expected String ASCII: Exp TTI Not Configured

Optics Status

Optics Type: DWDMWavelength Info: C-Band, MSA ITU Channel=15, Frequency=195.40THz, Wavelength=1534.250nm TX Power = 1.04 dBm RX Power = -5.33 dBm RX LOS Threshold = -16.00 dBm

TDC InfoTDC Not Supported on the Plim or

TDC InfoOperational Mode: AUTOStatus : LOCKEDDispersion Setting : 0 ps/nm


Performance Monitoring Over TimeCurrent Interval

Past Intervals

show controllers dwdm 0/15/0/0 pm history fecPort dwdm0/15/0/0

g709 FEC in the current interval [ 1:30:00 - 01:31:21 Mon Jun 28 2010]EC-BITS : 238922 Threshold : 0 TCA(enable) : NOUC-WORDS : 0 Threshold : 0 TCA(enable) : NO

g709 FEC in interval 1 [ 1:15:00 - 1:30:00 Mon Jun 28 2010]EC-BITS : 2254454 UC-WORDS : 0






APS channel is used for alarm signaling


SR porton Router

Trans-ponder

FEC

DWDM

BER Out

Packet Loss LOF

FEC

Standard Protection

Working Path Switch-over

ProtectedPath WDM

porton Router

FEC

DWDM

BER Out

Packet Loss

FEC

ProtectionTrigger

Near-hitless switch

Proactive Protection

Working Path Protected Path


Protection Type Fault Type

Convergence Time (ms)

Highest Lowest Average

Proactive Optical Switch 11.50 11.18 11.37

Proactive Noise Injection 0.02 0 0.0Proactive Fiber Pull (Tx) 25.48 0 12,39

Proactive PMD-Injection 0.08 0 0.02Standard Optical Switch 11,54 11,37 11,43

Standard Noise Injection 7404 1193 4305Standard Fiber Pull (Tx) 25.93 12.50 20.19

Standard PMD-Injection 129.62 122.51 125.90


RP/0/0/CPU0:router(config)# controller dwdm 0/0/0/0

proactive

log signal /disk0:/logfile1 /* start logging to the file

proactive trigger threshold 5 3 /* BER = 5E-3, window=30ms

proactive trigger window 100 /* changed to 100ms

proactive revert threshold 4 3

commit

RP/0/0/CPU0:router# sh controller dwdm 0/0/0/0 proactive

Proactive protection status: ON/OFF

Proactive protection state: Normal - interface is active

Inputs affecting proactive protection state:

Transport Admin State : IS/OOS/OOS-MT

Trigger Threshold: 3e-4

Revert Threshold: 5e-5

Trigger Integration Window: 100ms

Revert Integration Window: 2000ms

Received APS: 0x00 (No Request)

Transmitted APS request: 0x0A (Signal Degrade)


1. A-Z Provisioning sets congruent threshold for pre-FEC BER2. Whenever a degrade is detected, i.e. the BER pre-FEC threshold is crossed at REGEN, REGEN propagates a Degrade indication forward and backward3. FRR detects the degrade indication and switch

RegenA

RegenB

Threshold crossed at a regen, causing a signal towards upstream and downstream routers

Activates L3 switching

signal

signalRouter-A Router-

B


– FRR decides protection path ahead of a failure– Can be wrong w/o SRLG data

What appears diverse in L2/L3 may not be diverse in L1– Manual SRLG entry is error prone and not up to date– SRLGs can be mined from the Optical layer and fed to IP layer

!1 2 3

SRLG={1,2,3}

Router topology

Optical topology

Enhance network resilience w/o error-prone manual work


Alarm

EMS

TranspondersIn Shelves

TL-1/CORBA

DWDM Main Controller

w/ Info Model Database

Config

Traditional Network

Transponder Representation in Info Model

Comm Inside the NE (IPC)

Alarm

EMS

DWDM Router Interfaces

TL-1/CORBA/XML

Config

DWDM Main Controller

w/ Info Model Database

Virtual Transponder Representation in Info Model

LMP

IPoDWDM: Can Be Managed w/out Significant Changes


CESNET 40G IPoDWDM Testing 2009G.652

BalancedG.655


Requirements:„Big box” – multichassis OC-768QoSMulticastIPv6nx10GigE, not same as 40G


40G 613km ‘alien-wavelength’over existing Siemens Surpass hiT 7500

DPSK+ 40G

other NRZ 10G channelsQ Margin = 6.25 dBPMD < 2.3 psCD < +/- 700 ps/nmOSNR > 7.4 dB


40GE and 100GE


• IEEE 802.3ba D3.0 released (It’s a DONE DEAL)Standard Update


High Speed Ethernet Standard Interfaces


High Speed Ethernet – Not To Exceed PricingEstimated Not to Exceed List Prices (Industry-wide)


Historical Adoption of High Speed Ethernet

Cisco Confidential 46© 2010 Cisco and/or its affiliates. All rights reserved.All units are in millimeters and round numbers

Xenpak X2 XFP SFP+ QSFP CXP*

115

3677

1336 18

70

18

70 m

ax

~24

TBD56

CFP82

140

E-interface:84 pinsTx: 12x10GRx: 12x10G E-interface:

70 pins4x3.125G

E-interface:70 pins4x3.125G

30 pins1x10G

20 pins1x10G

E-interface:148 pins4x10G (XLAUI)10x10G (CAUI)

38 pins4x10G

High-Speed Transceivers Form Factors


High-Speed Ethernet Transceiver Landscape40G/100G CFP 40G QSFP

Applications:Single Mode Fiber 10-40+KmMultimode Parallel FiberTwinax CopperConvertible in to 4x10GbE (SFP+)Power Consumption:Up to 8W @ 40GbEUp to 25W @ 100GbE

Applications:Multimode Parallel FiberTwinax CopperFuture (?) 10 Km Single Mode

Power Consumption:Up to 3.5W


CFP features a new concept known as the riding heat sink, in which the heat sink is attached to rails on the host card and “rides” on top of the CFP, which is flat topped.

High-Speed Ethernet Transceiver Landscape

CXP was created to satisfy the high-density requirements of the data center, targeting parallel interconnections for 12x QDR InfiniBand (120 Gbps), 100 GbE, and proprietary links between systems collocated in the same facility. The InfiniBand Trade Association is currently standardizing the CXP.

100GbE CFP requires“Riding HeatSink” SMF optimized

100GbE CXP MMF/Twinax optimized


1x 100GBE� Line-rate performance (100Gbps)� CFP optics (LR4)

FP-140 – Core & Peering @ 140 Gbps� 8 queues per port, ACL, Netflow

MSC-140 – High-speed edge @ 140Gbps� HQoS, 64,000 queues, 12,000 interfaces

Interface Module

Forwarding Processor


20102010 20112011 20122012

NG Router Hardware Roadmap

20092009

CRS40G

per slot

CRS140Gper slot

CRS400Gper slot

ASR900080G

per slot

ASR9000200Gper slot

100GE 100GEOTU440GEOTU3e10GEOTU2e

10GEOTU2e

STM-256OTU3

10GEOTU2e

100GE40GE

STM-256

100GE/40GEOTU

7600 ES+40G

per slot


• Catalyst 6500: 4-port 40G module80Gbps/60Mpps, CFP transceivers16x 10GE version also available

• Nexus 7000: 6-port 40G module240Gbps/120Mpps, QSFP transceivers (focused on DC distances)

• Nexus 7000: 2-port 100G module (40/100)200Gbps/120Mpps, CFP transceivers (focused on wide-area distances)

Target FCS CY11

Data Center


• PM – QPSK – Pol Mux Quadrature Phase Shift Keying• Increase distances utilizing Cisco Advanced FEC• Advanced signal processing to address:

CD CompensationPMD MitigationSingle Channel Non-linear impairment mitigation

• To be implemented on both router interfaces and transport NEs

4 x50GADC

SignalProcessing

PIN

PIN

PIN

PIN

Laser

90°Hybrid

90°Hybrid

i0q0i1q1

Optical Linear System

Laser

25Gb/sQPSK1 Modulator

25Gb/s100 Gb/s = 25 Gbaud

25Gb/sQPSK2 Modulator

25Gb/s

In-phase

Quadrature

In-phase

Quadrature


• Higher data rates200Gig, 400Gig, 1T? Research for CRS, ASR, Nexus, ONS

• Need to investigate other modulation techniquesPM-16QAM, PM-64QAM, …. or CO-OFDM ?

• Need deeper look at FECAdvanced FECWhat other algorithms are there

• Need of intelligent DWDM layerFlex spectrumControl planeAdvanced operations, troubleshooting and protection mechanisms


• Motivations for IP NGN – traffic growth• Trends is IP/MPLS Core Design – cost reduction• Moore’s Law – making routers cheaper• IPoDWDM and proactive protection• 40GE,100GE and Beyond

Thank you.


Registrujte se za Cisco Live Networkers u Londonu ili Bahreinu!Više informacija na:http://www.ciscolive.com/

ipodwdm and ip ngn core design - cisco.com · the quality of ip ngn design •hierarchy (p is...

Documents