Page 1eav

Metropolitan IP Transport Networks

Eric A. Voit Distinguished Member of Technical Staff

TechnologyVerizon

November 8th, 2000

Copyright Verizon Communications, 2000. All Rights Reserved

Disclaimer: The views expressed herein are those of the author, and do not necessarily reflect the position of Verizon.

Note View Slides in PowerPoint

‘Slide Show’ format

Page 2eav

Agenda

• Economic drivers and architectures for Metropolitan IP Networks

• The importance of efficient facilities utilization

• More on IP QoS

Page 3eav

Long Live IP• IP divorces applications from transport

– Value added services will be IP based

– IP allows services to be delivered independently of the data link technology

– Customers want to exploit price-performance curves of access technologies while having each of these technologies inter-operate

• IP QoS has arrived– IP QoS mechanisms can be applied to specific applications of individual customers

– IP QoS is superior to ATM QoS for IP transport

• IP will be used for networks, ATM remains a viable link layer technology– DLECs currently provide integrated IP services using ATM transport. As lower cost

link layer technologies are deployed (such as Gigabit Ethernet/Fiber), DLECs will abandon ATM.

Page 4eav

Access Router (AR)– Provides local (End Office) access to the IP network

– Supports a variety of access technologies (e.g. ATM (DSL), Fast Ethernet, FR)

– Ensures the proper treatment of traffic based on an individual customer interface

– Off the shelf equipment

WireCenter

Access

Interconnection Router (IR)– Provides interconnection with external IP networks

– Ensures the proper treatment of traffic to meet QoS and Customer VPN needs

– Off the shelf equipment (potentially uses the same platform as the Router)

"Peer"ISP

Interconnection

Enterprise

Existing(Layer 2)

SP

PortalISP

Router (R)– Provides efficient IP transport between nodes of the network

– Ensures the proper treatment of traffic to meet QoS and Customer VPN needs

– AR and IR must also perform the functions of the Router

– Off the shelf equipment

Routing

DLEC MetroIP Transport Network

R

R

AR

AR

AR

IR

Review of Router Types and Business Segments

Page 5eav

A New Yorker’s View of the World

Page 6eav

ISP Networks AOL UUNET SBC GTE-I Sprint Yahoo

DLEC Metro IP Application Services

ISP Application Services

DLEC Metro IP NetworkRR

R RR R R RR R

R RR

R R R RR R

Residence Business

V/ I PGK

WebHosting

Chat eMail

IRIR

ARR

RAR AR

DSLAM LaserRack Ethernet

NIDATU-R

V/ IPGW

VideoCache

WebCache

AAASatelliteDownlink

SoftSwitchFirewall

A DLEC View of the IP World

Application

Transport

Page 7eav

Local

• Very high bandwidth• Low capital per installation• Limited local operations• Security / Network demarcation

Centralized

• Control Systems• Quickly changing data• LATA wide resources• Expensive platforms

Off Net

• Low volume services• Minimal QoS needs• World wide customer base• Embedded applications

Distributed

• High bandwidth

• Low Latency

• Local PSTN interconnection

Efficient Design ofLayer 1 & 2 Transport Network is Critical

Network Topology and Server Placement

RR

RR

AR

V/ IPGW

VideoCache

WebCache

AAAV/ IPGK

SoftSwitchemail Web

HostChat

RR

IR

RR

SatelliteDownlink Firewall

LaserRack

EthernetNID

DSLAM

ATU-RLocal Servers

High Geographic SignificanceRemote ServersLow Geographic Significance

Page 8eav

SCOPE

DLECHostedApplications

ServiceUser

ISP Networks &Applications

DLEC MetroIP Network

Scope of Metropolitan IP Network

LatencyJitterPacket LossPacket SequenceBandwidthSecurityAvailability

Service Level Agreements (SLA) at IP Network Interfaces are Negotiated based on Application Specific Needs

V/IPGW

Chat Video

WebHosting

WebCache

eMail

Application SLA

Application SLA

Page 9eav

End-to-End Service Level Agreements

W ire Center W ire Center

IP Transport Netw ork

P S TN P S TN

ARVoice / IPGatew ay

AR Voice / IPGatew ay

ApplicationProvider

TransportProvider

ApplicationProvider

50 M S 80 M il l iS e conds (M S ) 50 M S

W ire Center W ire Center

IP Transport Netw ork

P S TN P S TN

ARVoice / IPGatew ay

AR Voice / IPGatew ay

ApplicationProvider

TransportProvider

ApplicationProvider

50 M S 80 M il l iS e conds (M S ) 50 M S

• An application layer budget needs to be partitioned between sub-systems

• SLAs need to be measurable and actionable

Page 10eav

AR Integrates Diverse Transport Technologies

• Access Router acts as universal IP edge device for diverse customer access methods– Inexpensive equipment from a highly competitive evolving marketplace

• Access technologies only supported as new services demand them (incremental roll-out)

• Sharing the IP WAN infrastructure allows AR to push further to the edge

Central Office

Business

Wire Center

Residence

ATU-RDSLAMDSL

ModemBank

ModemPSTN Circuit

Fast & Gig Ethernet / FiberLaserRack

SONETADM

SMDS

Packet over SONET

Frame Relay

ATM

Customer

R

WASLTower

WASLBroadband Wireless

ARAny Link Layer (2) Technology

Page 11eav

C entra l O ffice

AR

OC3

CBQDevice

CustomerPC

CustomerIP Phone

Customer

PC

ExternalATU-R

NICATU-R

DSLAM

xDSL

ADSL

Dozens of Twisted PairOne Fiber

Layer 2 identity based

on VPI/VCI

OpticalNID

Pedestal

OpticalEthernet

Mux

Customer

PC

WDM10/100

BaseFX

GBE

Dozens of Feeder FibersHundreds of Drop Fibers

Several Fibers (redundancy)

The Key to Access ScalabilityDecoupling Logical and Physical Link Layer Termination

• Router Port 1– Multiple ATM VCs from a DSLAM

• Router Port 2– Ethernet access ‘Channelized’ for Fiber to the home

• Router Traffic engineering is based on offered load and the number of logical connections, not physical port limitations

Page 12eav

Wire Center N

Wire Center 1Local Area

AR

AR

AR

ARAR

AR RR

AR

ARAR

AR

AR

AR

AR

IRIR

Metropolitan IP Routing Topology

• Logical– Redundancy and failover

supported by proven routing protocols & implementations

– Opportunity to efficiently route local and InterLATA IP traffic

Wire Center N

Wire Center 1Local Area

ARARAR

AR

AR

ARAR

AR

AR

AR

AR

IR

ATM

Packetover

SONET

GigabitEthernet

overFiber

ATM

AR

AR

IR

RR

• Physical– Router connections

engineered using the most efficient / expedient transport alternatives

– Can change without impacting logical design

Engineering Decisions

Page 13eav

Vertical Services Domain

VoiceOther

OSPF

OSPF

R

R

R

Video

Vertical Services Domain

VoiceVideo

OSPF

OSPF

OS

PF

OSPF

OSPF

OSPF

OS

PF

OSPF

OSPF

OSPF

R

R

R

R

OSPF

R

R

RR

BG

P4

BG

P4

BGP4

BGP4

Enterprise

IR

IR

"Peer"ISP

Existing(Layer 2)

SP

PortalISP

OSPF

ARAR

AR

AR

AR

AR

AR

AR

AR

AR

SRAR

AR

RR

Metro Boundary

Local Area 4Local Area 1

Local Area 3

LocalArea 2

Routing and Interconnection Logical Topology

• Metro is built from multiple local areas

• Local areas are connected to ensure redundancy and performance

• Content services can be centralized, or distributed to the edge

• Interconnection with ISPs is centralized to minimize operations complexity

Page 14eav

Network Topology and the Importance of Efficient Facilities Utilization

Page 15eav

Which of the following statements are true:

• IP, ATM, and SONET layers exist as independent aggregations of signaling, transport, and operations protocols and equipment

• This layering results from historical accident

• For networks carrying IP, this layering is based on sound long term engineering principles & economics

1996-1999 Data Networks

Network Provider ATM / FR

Network Provider SONET

Network Provider Fiber

Customer IP LAN

Customer IP LAN

Page 16eav

Architectural Drawbacks for Local Data

• Can require more than 6x the SONET ring bandwidth of an IP/SONET connection

• Why? SONET was designed assuming a local PSTN switch that could choose a local trunk group. Star data architectures are unable to leverage this ability.

– The current architecture was developed when data volumes were low, switches were expensive, and operations procedures and expertise was evolving

– The current architecture is best when data traffic isn’t local

• At some point, local data volumes become large enough for the economics to favor the deployment of data switches and routers closer to the edge of the network

– Any economic analysis would have to measure the amount and characteristics of the local data to determine the opportunity for savings

– The analysis would then focus on facilities cost versus switch placement, operations, and maintenance costs

– An unlikely analysis conclusion would be to simply distribute more switches and routers across our existing SONET …

Layer 2 Switch

Example: Two customer routers, both connected to a Layer 2 Cloud via PVCs

x2

S

RR

Customer owned Routers

Page 17eav

R

SONET

SADM

ADM

R

R

PVC or SVCPVC or S

VC

ADM

ADM

ADM

or

CustomerCustomer

Network Provider

x2 Bandwidth

R

SONET

SR

R

PVC or SVC

PVC or SVC

PVC or SVC

ADM

ADM

ADM

or

CustomerCustomer

Network ProviderS

ADM

ADM

NetworkProvider

x3 Bandwidth

Installing a Switch (or Router) on Every SONET Ring Doesn’t Completely Solve the Problem

• If a switch or router was placed on every SONET ring, you would still double the data bandwidth required between two local offices (compared to a direct SONET connection)*

• Adding a second switch on the ring in some cases will TRIPLE the SONET bandwidth required

• Optimizing SONET utilization (for local data) requires switching capability in every C.O.– Optimizing SONET costs doesn’t mean you have optimized total service delivery cost– Today operational, service, and complete network topology roadblocks hinder such a network configuration– Architectures which address these problems are emerging– Price points for equipment and operations are changing, and are different than when Fast Packet services were

first deployed

* This example intentionally ignores the benefits of multiplexing traffic to multiple destinations, both local & remote.

Page 18eav

SONET also ‘Wastes’ Bandwidth

C.O.

Central Office

'Tandem'C.O.

C.O.

OC-48

ADM

ADM

ADMADM

AR

ARR

AR

OC3

OC3

OC3

OC3

• This fiber run is carrying an OC-12 worth of ‘wasted’ bandwidth. With traditional SONET, you cannot use this available capacity to transport low priority traffic.

(Note: this is not PoS interface protocol issue.)

Logical

Phys

ical

Page 19eav

Gigabit Ethernet & 10GBE over Fiber

Enabling Factors:

• Dark fiber

• New Routers– ASIC & FPGA

– Low latency

– Big backplanes (50+ GB/s)

– Line Speed QoS• Layer 3/4

– 802.3ad

– Interrupt driven failure recovery

C.O.

Central Office

'Tandem'C.O.

C.O.

AR

ARR

AR

When is the cost of leased fiber (over diverse paths) lower than SONET?

Page 20eav

CentralOffice

CentralOffice

RFiber

Frame

Access& IOFLaserRack

IP/Gigabit Ethernet/Fiber

RFiberFrame

Access& IOFLaserRack 80

2.3a

d

802.

3ad

Ethernet / Fiber Organizational Interfaces

• Different Organizations would manage the Routers and the Transport Layer• Different skill sets for Operations• Useful life of long haul Lasers is longer than life of Router• Ethernet provides a simple interface within the C.O.• Same network interface for Router to Router and Router to Customer

communications• DWDM can be deployed to transport multiple Ethernet handoffs from the Routers• Regulatory boundaries

Page 21eav

C.O.

Central Office

'Tandem'C.O.

C.O.

AR

ARR

AR

to DLECBackbone

DLEC Local Topology & Existing Tariffs

Verizon Transport Alternatives

• ATM – DS3

– OC3

• SONET– DS3, OC3 on shared ring

– Dedicated OC48 ring (OC12 drops to AR)

• Dark Fiber

DLEC Local Topology

Airline perimeter around the four C.O.s

Urban(10 miles)

Rural(50 miles)

Page 22eav

Verizon Recurring Monthly Tariffs (FCC Tariff #1, 3 year commitment)

4 fiber strands = $1200 per mile

FCC#1 7.5.10

DS3

Distance to ATM hub Price

0-5 miles $2,460 5-25 miles $3,360 25-50 miles $5,645

FCC#1 16-6-1 (A)

ISAN (Shared Ring)Per IEF connection $636 0-3 miles $2,5004-20 miles $3,700 20+ miles $5,500

FCC#1 7-9-20 & 7-9-21 (shared)

OC12

Unavailable

not offered

Dedicated OC48(w/ OC12 Drops at Node)

Ring connectivity: $7,900 Per mile charge: $639

FCC#1 7-5-19 (dedicated)

OC192

Unavailable

technical issues

Unavailable

not offered

OC3

Distance to ATM hub Price

0-5 miles $4,965 5-25 miles $6,610 25-50 miles $8,790

FCC#1 16-6-1 (A)

ISAN (Shared Ring)Per IEF connection $2,493 0-3 miles $5,6004-20 miles $7,800 20+ miles $10,200

FCC#1 7-9-20 & 7-9-21 (shared)

ATM(UNI to Cell Relay Cloud)

Dark Fiber(point to point)

SONET(point to point)

* Per VC Charges excluded

*

Page 23eav

More on IP QoS

Page 24eav

Why IP QoS is Superior to IP/ATM QoS

• Applications talk IP

• IP routers can now identify IP application layer traffic flows, and prioritize them across the LAN (i.e. QoS)

• Supporting IP flow QoS in the WAN is now becoming viable

• If IP QoS is deployable, having an intervening ATM QoS abstraction is redundant, unnecessarily restrictive, and costly– QoS prioritization is done on the AR

– Eliminates the need for the different types of ATM pipes

– Eliminates traffic management and operational complexities of different pipes

– Minimizes troubleshooting between the IP and ATM layers

• Managing QoS is now something customers can outsource to the network. They don’t have to pre-sort their IP traffic into different types of ATM QoS differentiated VC pipes

Page 25eav

ATM Switch

CustomerPC

EthernetATU-R

DSLAM

ADSL

OC3

DS3

UBR

UBR

UBR

CBR

VBR-RT

IP

ATM

ATM

ATM

ATM

1 FTP Packet at 1500 bytes

32 ATM Cells at 53 bytes

ATM

ATM

ATM

ATM

ATM

320 ATM Cells(5 dropped = 265 bytes)

ATM

ATM

ATM

ATM

ATM

960 ATM Cells(15 dropped = 795 bytes)

ATM

Router

CustomerPC

EthernetATU-R

AR/DSLAM

IP

ATM

ATM

ATM

ATM

1 FTP Packet at 1500 bytes

32 ATM Cells at 53 bytes

ATM

10 IP Packets(1 dropped = 1500 bytes)

30 IP Packets(1 dropped = 1500 bytes)

ADSLUBR

DS3

OC3

IP

IP

IP

IP

IP

IP

IP

IP

IP

IP

ATMCBR_

ATMCBR_

Moderate ATM Cell Loss Can Induce Disproportionate IP Packet Loss

4% of Cells 4% of Packets

40% of Packets * 4% of Packets

DLEC discards:

IP/ATM IP

Subscriber experiences loss of: