1

Resource Optimization in Hybrid Core Networks with 100G Links

Malathi VeeraraghavanUniversity of Virginia

[Collaboration with Admela Jukan]

Date: Sep. 28, 2009

Outline• Problem statement• Hybrid network architecture• Traffic monitoring/characterization • Traffic engineering• Network engineering• Traffic analysis & simulations• DOE testbed• Workplan

2

Big picture question• As

– link rates reach 100Gbps– switching capacities reach tens of

Tb/s• Do we need a new backbone

network architecture?

3

Hybrid node architecture Unfolded view

Inputinterface

s

Outputinterfaces

WDM switch100Tbps

Layer-3 (IP) router (1Tbps)

Data plane

SNMP MIB+agent

sRouting protocol

Signaling (provisionin

g)protocol

Administrative interface (CLI, TL1)

Node controller

Hybrid node

Ethernetcontrol-planeport

Carrier Ethernet

switch (10Tbps)

Hybrid network architecture

5

Shared single core poolof N fibers

K circuits: IP-routed partitionN-K: Dynamic-circuit partition Hybrid

Node

HybridNode

HybridNode

HybridNode

HybridNode

Hybrid Traffic engineering (TE) system Hybrid Network Engineering (NE) system“put the traffic where the bandwidth is” “put the bandwidth where the traffic is”

REF: Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking.Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf

Modify routing metrics and/orwrite routing table entries

Request dynamic circuitsetup/releaseTraffic monitoring/

characterization systemDOE-implemented control-plane

software systems

Obtain data

REF

Traffic monitoring/characterization system

6

Shared single core poolof N fibers

K circuits: IP-routed partitionN-K: Dynamic-circuit partition Hybrid

Node

HybridNode

HybridNode

HybridNode

HybridNode

Hybrid Traffic engineering (TE) system Hybrid Network Engineering (NE) system

Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking.Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf

Traffic monitoring/ characterization system

DOE-implemented control-plane software systemsTraffic monitoring/characterization system

• Reads parameters necessary only for TE/NE applications• Characterizes traffic matrix• Not itself a general-purpose monitoring system such as PerfSONAR but could interface with such systems to obtain data

Hybrid Traffic Engineering system

7

Shared single core poolof N fibers

K circuits: IP-routed partitionN-K: Dynamic-circuit partition Hybrid

Node

HybridNode

HybridNode

HybridNode

HybridNode

Hybrid Traffic engineering (TE) system Hybrid Network Engineering (NE) system

Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking.Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf

Traffic monitoring/ characterization system

DOE-implemented control-plane software systemsHybrid Traffic Engineering (TE) system

• Obtains data from Traffic monitoring/characterization system• Computes optimal routes for load balancing• Issues CLI commands to hybrid nodes to modify routing metrics and/or write routing table entries

Hybrid network engineering system

8

Shared single core poolof N fibers

K circuits: IP-routed partitionN-K: Dynamic-circuit partition Hybrid

Node

HybridNode

HybridNode

HybridNode

HybridNode

Hybrid Traffic engineering (TE) system Hybrid Network Engineering (NE) system

Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking.Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf

Traffic monitoring/ characterization system

DOE-implemented control-plane software systems

Hybrid Network Engineering (NE) system • Obtains data from Traffic monitoring/characterization system and DOE-implemented control-plane software• Determines if thresholds are crossed to trigger setup/release of dynamic circuits• If triggered, sends request for dynamic circuit setup/release to DOE-implemented control-plane software• Commands Hybrid TE system to make routing table updates

What is today’s backbone network architecture

9

• Two separate networks (separate nodes; separate links)– IP-routed service (black links)– SDN for dynamic-circuit service (blue links)

Summer 09 ESnet map: http://www.es.net/pub/maps/current.pdfIP routerSDN switch

Nodes and wide-area links• IP routed network

– 16 routers (MX960 or M320)– 20 inter-city (WAN) links – 10GbE

• SDN network– 17 SDN switches (MX 960)

• two in New York, two in Chicago– 29 inter-city (WAN) links – 10 GbE

• Metro rings– SUNN, CHIC, NEWY

10

Future hybrid network:Part of the ESnet topology

• For simplicity, let us drop the WDM switch from our hybrid node and just have one connection-oriented switch (e.g., carrier Ethernet, MPLS, WDM, SONET) 11

ALBU

DENVPNWG

SUNN ELPA

Connection-Oriented (CO) switch

100 GbE

IP Router

10GbE

LANLSNLA

Hybrid node

PNNL

Hybrid architecture

12

SDN switchIP router

ALBU

SDN switch IP router

DENV

CO switch IP router

ALBU

CO switch IP router

DENV

100 GbE10 GbE 10 GbE

NREL

dynamic circuit

SNLA10 GbE

1 GbE

1 GbE

SNLA

10 GbE

LANL100 GbE

Current: Separate Future: Hybrid

NREL 10 GbE

LANL

What rate?

ComparisonCurrent architecture Hybrid (future) architectureSeparate inter-city (backbone) links – Two or more 10GbE

One inter-city (backbone) link- 100 GbE

Most enterprise access links terminate only on IP router – 10GbE or 1GbE(some on both IP router and SDN switch)

Enterprise access links terminate only on CO switch – 10 GbE or 100 GbE

Backbone links are 10GbE (same as some enterprise access links)

Can we make IP router-to-router circuits of lower rate?

13

Link loads are low now; nevertheless 10Gb/s links are required to avoid backbone links from becoming bottlenecks

Questions• Backbone router to backbone router

circuits:– bandwidth-limited?

• not mandatory to limit circuit bandwidth if CO switch is MPLS or carrier Ethernet

– bottleneck link implication on long-flow TCP throughput

– if so, what rate?• Access circuits (enterprise-to-

backbone):– what rate?

14

IP datagram flow in future hybrid network

15

ALBU

DENVPNWG

SUNN ELPA

Connection-Oriented (CO) switch

100 GbE

IP Router

10GbE

LANLSNLA

Hybrid node

PNNL

Implication• Packet forwarding:

– CO switches handle packets at intermediate nodes– Backbone IP routers involved only at source and

destination PoPs– “Cut-through” paths

• By reducing number of packets handled by IP router:– its’ switching capacity can be smaller– access and backbone circuit rates between

access/backbone routers can be lower – low rates will not affect short flows, but will impact

long flows– can long flows can be redirected to CO network?

16

Hybrid architecturefeasibility study

• Traffic analysis• Simulations

– to enable planning of node size (port speeds, switching capacities)

17

Traffic analysis• Analyze Netflow data

– characterize “short” and “long” flows• flow inter-arrival time distributions• flow duration distributions

– need to characterize lengths of silences between long flows generated by an enterprise

– determine what applications generate “long” flows

18

Simulations• Use Netflow data-validated input traffic model

for ESnet• Create a simulation model of an alternate

“hybrid” ESnet, one in which inter-router links are of lower rates than in deployed network but used only for short flows

• Long flows are directed to CO network with dynamic circuit setup between enterprise routers

• Obtain performance metrics– short flows– long flows

19

SNMP traffic analysis • Obtain SNMP link traces for

backbone and access links• Are some links that are not on

many shortest paths loaded at lighter levels than others?

• Time-of-day dependencies?• Correlations? • Stationarity?

20

DOE-provided testbed• Demonstrate operation of hybrid

networks on DOE-provided testbed with 3 to 4 nodes.– Hybrid nodes (provided to us):

• IP router (layer 3) + SDN switch (layer 2.5, 2 or 1)• DOE-implemented control-/management-plane

software– OSCARS, PerfSONAR

– Implemented by us:• Traffic characterization software• Traffic/networking engineering management

systems21

Work plan• Year 1: Architecture and Analysis• Year 2: Algorithm design and

software implementation• Year 3: Prototyping on DOE-

provided testbed

22

Requests to ESnet• Netflow data • Raw SNMP data files • RIBs (BGP and IGP) and IGP metrics• Can a full-mesh of MPLS tunnels - without bandwidth

allocations – be created on existing ESnet?– Simplifies the process of generating traffic matrices (LSP SNMP data)

• Are there experimental PCs at PoPs for researcher use?– Virtual machines?– PCs connected to both IP router and SDN switch?– Run BWdetail (data-plane experiments)

• Is http://netinfo1.es.net to be maintained? If so, the Public Access link gives error messages. Will Level2 access be useful to us?

23