Super Jumbo Frames Super Jumbo Frames 16-64k path MTU16-64k path MTU

performance analysisperformance analysis

BCNET-Netera

Advanced Networks Conference

2006

Network RRX

What are Super Jumbo Frames ?What are Super Jumbo Frames ?

PREMAC/

LLCIP Hdr TCP Hdr FCSGapInterFrame

Packet (MTU)

Payload Data

•SJF Payload Data > 9000 bytes

•IPv4 Header 16 bit limit

•65,535 minus header

•IPv6 Extension 32 bit limit

•4,294,967,295 minus header

Network RRX

Why do SJF?Why do SJF?

•Moores Law starting to plateau

•HPC community going to larger interconnect solution frames

•Main backbones scaling up to 100G

Network RRX

SJF Hardware?SJF Hardware?

•Enterasys ER16 Series Routers

•MTU = 65442 bytes

•Spirent Communications –AX/4000

•MTU = 64724 bytes

Network RRX

Inter Frame Gap (IFG)Inter Frame Gap (IFG)

•IFG - time between end of one packet and beginning of next

•IFG Inflation - when bigger payload causes IFG increase

•Minimum IFG implies max network performance

PREMAC/

LLCIP Hdr TCP Hdr FCS

InterFrame Gap (IFG)

Payload DataGapInterFrame

Network RRX

SetupSetup

16k Node

9k lightpath

9k Node

multi port

Xnet 1G 16kNetwork Diag swEnterasys router

Xnet 10G 64k/9kSpirent softwareswitch/router

Xnet 1G 64kSpirent softwareEnterasys router

Xnet 1G 16kDFS App swEnterasys router

10G IF

Spirent

Netera Switch

AX/4000Chassis

Generators

Generators Analyzers

16k Node

LC SMLC SM

SX SC MM1310nm

linux

9k Nodew2k

SX SC MM850nm

SX SC MM850nm

SX SC MM850nm

CRS - 1X8

MM/SC/SXX/Y = ?

MM/SC/SXX/Y = ?

MM/SC/SXX/Y = 8/9

MM/SC/SXX/Y = 9/10

MM/SCX/Y = ?/?

1G IFSpirent Spirent

SX SC MM850nm

SX SC MM850nm

CRS -1ER16

GBIC SCMM 850nm

GBIC SCMM 850nm

GBIC SCMM 850nm

9k Node

w2k

10G LR XFPSM 1310nm10G LR XFP

SM 1310nm

LC SMLC SM

Xnet 1G 16kNetwork Diag swEnterasys router

Xnet 10G 64k/9kSpirent softwareswitch/router

Xnet 1G 64k SpirentsoftwareEnterasys router

Xnet 1G 16kDFS App swEnterasys router

MM/SCX/Y = 11/10

SVGA cable ... ?SVGA cable ... ?

Copper RJ45

Copper RJ45

MM/SC/SX1G 850 nm

MM/SC/SX1G 850 nm

SX SC MM850nm

routed

NIC SC MM850nm

NIC SC MM850nm

NIC SC MM850nm

MM/SC/SX1G 850 nm

MM/SC/SX1G 850 nm

HDLC/PPP encapasulation ends

HDXc 8th flr

HDXc

SCInetNOC

ONS

VanTX

ONS

VicTX

ONS

Sea01 DWDM

DWDM

ONS

Sea01

10G 10G

1G

10G

10G IF

Spirent

Diag SessionTest results

Analyzers

10G SC SM1310nm

HDLC/PPP encapasulation starts

NortelBooth540

native ethernet

10G SC SM1310nm

Network RRX

ResultsResults

25

50

75

100

%

105104103102

MSS (bytes)

9000 64000160001500

thro

ug

hp

ut

CA*net

SC’ 05 - 25 s IFG

Abilene

Network RRX

IFG InflationIFG Inflation

IFG

s

PKT (s)

PKT + IFG0

IFG0 = 15.94 = 0.364

037.5 75

37.5

75

Abilene Avg IFG with Std Dev

0.75

IF

G

s

IFG0 = 12.99

10

17.5

25

CA*net Avg IFG with Std Dev

PKT (s)

37.5 750.75

PKT + IFG0

= 0.095 CA*net

0.095

Abilene

0.364

Network RRX

SimulationSimulation

0.5

1

1.5

2

dn

MSS (bytes)105104103102 106 107

9000

16000

64000

512

640000 2.56E6 6.4E6

n1/2 n1/2 n1/2 n1/2

0.78

1.56

n1/2 MSS1 G 259510 G 2528040 G 100900100 G 252200

CA*net Experimental AvgAbilene Experimental Avg

Linear IFG FitLinear IFG Extrapolation

100 G=0.00229

10 G=0.0229

1 G=0.229

40 G=0.00574

Reference Experiment

optimized

non-optimized

64000 640000 2.56M 6.4M

Network RRX

SJF Memory IssuesSJF Memory Issues

•Enterasys ER16 uses direct memory access (DMA) to a common main memory system

•Ethernet integrated circuits generally have internal 64 or 128 kilobyte buffers

Network RRX

SJF ContributorsSJF Contributors•E. Paul Love (Xnet co-chair)

•Paul Daspit (Xnet co-chair)

•Greg Goddard (Spirent)

•Bob Anderson (Spirent)

•Randy Ottman (Enterasys )

•Arnold Sodder (Enterasys )

•Issa Eid (Enterasys )

•Gary Finley (Netera)

•Loki Jorgenson (Apparent Networks)

•Martin Siegert (SFU-WestGRID)

•Lixin Liu (SFU)

•Peter Van Epp (SFU)

•Bryan Caron (UofAlberta)

•Brian Corrie (SFU-IRMACS)

•John Moore (NCSU)

•Kevin Walsh (SDSC)

•Paul Schopis (ITEC-Ohio)

Network RRX

End of Presentation