gridpp collaboration meeting may 2002 r. hughes-jones manchester 1 networking in under 30 minutes !...

GridPP Collaboration Meeting May 2002R. Hughes-Jones Manchester

1

Networking in Under 30 Minutes !

Richard Hughes-Jones, University of Manchester


2

E-science core projectMB - NG

Project to investigate and pilot: end-to-end traffic engineering and management over multiple administrative

domains – MPLS in core diffserv at the edges.

Managed bandwidth and Quality-of-Service provision. (Robin T)

High performance high bandwidth data transfers. (Richard HJ)

Demonstrate end-to-end network services to CERN using Dante EU-DataGrid and to the US DataTAG.

Partners:CISCO, CLRC, Manchester, UCL, UKERNA plus Lancaster and Southampton (IPv6)

Status: Project is running with people in post at Manchester and UCL.

Project Tasks have been defined and Detailed planning in progress

Kit list for the routers given to Cisco

Test PC ordered

UKERNA organising core network and access links – SJ4 10Gbit upgrade

Strong Links with GGF


3

MB – NG SuperJANET4 Development Network (22 Mar 02)

Gigabit Ethernet2.5 Gbit POS Access2.5 Gbit POS coreMPLS Admin. Domains

Dark Fiber (SSE)POS

WorldCom

SuperJANET4ProductionNetwork

LeedsSJ4 DevC-PoP

Warrington12416

SJ4 DevC-PoPLondon12416

UCLOSM-4GE-WAN-GBIC

OSM-1OC48-POS-SS

UCLOSM-4GE-WAN-GBIC

MCCOSM-4GE-WAN-GBIC

OSM-1OC48-POS-SS

MANOSM-4GE-WAN-GBIC

RALOSM-4GE-WAN-GBIC

OSM-1OC48-POS-SS

RALOSM-4GE-WAN-GBIC

OC48/POS-SR-SC

SJ4 DevULCC12016

OC48/POS-SR-SC

WorldCom

SJ4 DevC-PoP

Reading12416

OC48/POS-LR-SC

WorldCom ULCC

MB - NG


4

Defining Background Traffic

Regular traffic - constant size packet, regular spaced in time Poisson traffic - constant size, exponential spacing to form transient queues IETF traffic mix - different sizes and different probability of each size sent Play back of real traffic patterns generated from packet headers pre-recorded from suitable

points of the production network. This might include:

Video Conference traffic -> play back - rude/crude tools

UCL real conf playback tool

General traffic captured at edge of a site, e.g. Manchester

Do tests with a gen to see what gets dropped 0.5 Gbit typical peak UCL Web-bursty traffic – web mirror – wget Need to be able to reproduce traffic Statisticaly In general UDP best to understand the net

Consider UDP TCP flows Need ToS / QoS to be set How to control

Start Stop

measure load as function of time – links and Queues

Start and end numbers


5

Defining the Measurements (10) UDP round-trip latency vs packet size & Histograms

Sum of dt/dl transfer rates Router & stack performance Indication of network load & Switching / transport effects – spikes

UDP 1-way delay UDP throughput vs Packet size and transmit delay

Throughput behaviour Offered vs achieved throughput

UDP Packet Loss vs transmit rate and burst size Loss rate Packet loss distribution as function of time Buffer sizes in the path & Detect packet re-ordering

UDP Inter-frame Jitter as function of packet transmit spacing Indication of network load Behaviour of end system NICs

TCP round-trip latency vs Message size & Histograms Sum of dt/dl transfer rates Stack / protocol performance – detect Packet size dependencies

TCP throughput vs Message size and transmit delay Throughput behaviour cf UDP Packet loss distribution as function of time + Re-transmit rate

TCP throughput vs Window size / TCP tuning TCP throughput vs number of streams

Stream throughput – benefits & effect on the network Packet loss distribution as function of time + Re-transmit rate

TCP Protocol behaviour - tcptrace

Align Metrics with GGF/IETF


6

Defining the Measurements (11) TCP round-trip latency vs Message size

Sum of dt/dl transfer rates (Router performance) Stack / protocol performance Detect Packet size dependencies

TCP round-trip histograms Indication of network load (Switching / transport effects – spikes) Stack / protocol performance

TCP throughput vs Message size and transmit delay Throughput behaviour cf UDP Offered vs achieved throughput Packet loss distribution as function of time + Re-transmit rate Loss as func of pkt rate eg keep the data rate the same change pkt size – multi-streams

TCP throughput vs Window size / TCP tuning TCP throughput vs number of streams

Stream throughput - benefits Packet loss distribution as function of time + Re-transmit rate Effect on Network

TCP Protocol behaviour - tcptrace What are the “burst” lengths Effect of routers / end system NICs

All this for WRED Wt fair Qing data rate const and ch pkt size – chack how well the routers do the Qing

Align Metrics with GGF/IETF


7

The EU DataTAG project

EU Transatlantic Girid project. Status: Well under way – People in post, Link expected Jul 02 Partners: CERN/PPARC/INFN/UvA. IN2P3 sub-contractor US Partners: Caletch, ESnet, Abilene, PPDG, iVDGL … The main foci are:

Grid Network Research including:

• Provisioning (CERN)

• Investigations of high performance data transport (PPARC)

• End-to-end inter-domain QoS + BW / network resource reservation

• Bulk data transfer and monitoring (UvA)

Interoperability between Grids in Europe and the US

• PPDG, GriPhyN, DTF, iVDGL (USA)PPDG, GriPhyN, DTF, iVDGL (USA)


8

DataTAG Possible Configurationmulti-platform multi-vendor

AbileneAbileneESNETESNET

UKUKSuperJANET4

ITGARR-

B

ITGARR-

B

GEANT

NLSURFne

t

NLSURFne

t

2.5 Gbit PoS lambda

SLAC

Cisco

Juniper

Alcatel

Cisco

Juniper

AlcatelGigaSwitch

JuniperLight

Switch

GigaSwitch

Cisco6509

JuniperLight

Switch

CERN (Geneva)Starlight (Chicago)

Fermi


9

The SuperMicro P4DP6 Motherboard

Dual Xeon Prestonia (2cpu/die) 400 MHx Front side bus

Intel® E7500 Chipset 6 PCI-X slots 4 independent PCI buses Can select:

64 bit 66 MHz PCI 100 MHz PCI-X 133 MHz PCI-X

2 100 Mbit Ethernet Adaptec AIC-7899W dual channel SCSI UDMA/100 bus master/EIDE channels

data transfer rates of 100 MB/sec burst Collaboration:

Boston Ltd. (Watford) – SuperMicro Motherboards, CPUs, Intel GE NICsBrunel University – Peter Van SantenUniversity of Manchester – Richard Hughes-Jones


10

IntelPro1000 : P4DP6 64bit 66MHz PCI slot4

y = 0.0135x + 174.05

y = 0.0121x + 178.31

0

50

100

150

200

250

0 500 1000 1500 2000 2500 3000

Message length bytes

Late

ncy u

s

Latency & Throughput: Intel Pro/1000 on P4DP6

Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

Latency high but smooth Indicates Interrupt coalescence Slope 0.0136 us/byte, Expect:

PCI 0.00188

GigE 0.008

PCI 0.00188

0.0118 us/byte

Max throughput 950Mbit/s Some throughput drop for packets

>1000 bytes

www.hep.man.ac.uk/~rich/net/nic/GigEth_ tests_Boston.ppt

UDP IntelPro1000 : P4DP6 64bit 66MHz PCI slot4

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame us

Recv W

ire r

ate

Mb

its/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes


11

New External ConnectivityNew External Connectivity

6 * 155 Mbit links2.5Gbit line installedIP commodity peer in London Research traffic over 2.5G bitPeer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.


12

Connectivity to Europe : GeantConnectivity to Europe : GeantStart mid November 2001UKERNA switched off TEN-155 3 Dec 2001


13

ICFAMON Plot from DL to CERN for 18th Feb to 3rd Mar 2002

Connectivity to EuropeConnectivity to Europe

UK Dante Access link 2.5 Gbit POS

Remember 19th Oct to 1st Nov 2001155 Mbit Access link over loadedSustained rate 130 Mbit


14

Monitoring: US Traffic Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US

7 day periods 1 hour averages

Weekend-Before Weekday-After

Weekday-Before

14 Jan 2002 (800Mbit/s)peak 86% of total 930 Mbit

17 Jan 2002Peering altered 22 Jan

22 Jan 2002 Weed day peak 175 Mbit/s


15

Monitoring: US Traffic Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US7 Dec 2001 (900kbit/s) 29 Jan 2002 (175kbit/s)peak is 88% of total BW 930 Mbit10 minute averages 10 minute averages

7 days 1 hour averages


16

DataGrid Network Monitoring

Several tools in use – plugged into a coherent structure:

PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP, and NWS prediction engine

continuous tests for last few months to selected sites:

DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … The aims of monitoring for the Grid:

to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling

to identify fault conditions in the operation of the Grid

to understand the instantaneous, day-to-day, and month-by-month behaviour of the network – provide advice on configuration etc.

Network information published in LDAP schema – new self-defining Cost Function in development – collaboration with WP1 & WP2 Will be used by UK GridPP and e-science sites and non HEP WPs Links to the US


17

Local NetworkMonitoring

Store & Analysisof Data (Access)

Access to current and historic dataand metrics via the Web, i.e. WP7NM Pages, access to metric forecasts

Backend LDAP script to fetch metricsMonitor process to push metrics

localLDAPServer

Grid Application access viaLDAP Schema to- monitoring metrics; - location of monitoring data.

PingER(RIPE TTB)

iperfrTPLNWS

etc

LDAPSchema

Grid AppsGridFTP

Network Monitoring Architecture

Robin Tasker


18

Network Monitoring Components

Ping Netmon UDPmon iPerf Ripe

Cronscript

plot

Table

LDAP

raw

control Cronscript

controlCronscript

plot

Table

LDAP

raw plot

Table

LDAP

raw

WEB Display AnalysisGrid BrokerPredictions

Web I/f

Scheduler

Tool

Clients

LDAP

raw

LDAP

raw


19

Network Monitoring


20

Network Monitoring: Ping


21

Network Monitoring: Iperf (TCP)


22

HI Performance UDP Man - RAL Gigabit I/f

Man-RAL Gig 20 Apr 02

y = 0.0904x + 5271.8

y = 0.018x + 5409.1

0

1000

2000

3000

4000

5000

6000

7000

0 500 1000 1500 2000 2500 3000


Late

ncy u

s

Latency 5.2 mS Slope 0.0904 us/byte or 2 PCs expect:

PCI 0.00188

GigE 0.008

PCI 0.00188

Total 0.0118 us/byte

7 routers extra links3 GigE 0.024

2 2.5G PoS 0.0064

1 622Mbit 0.012

Total 0.0424

Structure seen: Period 88 bytes Variation 150 – 190 us

Max throughput 550Mbit/s Some throughput drop for

packets < 20 us spacing

UDP Man-RAL Gig 21 Apr 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40


Rec

v W

ire

rate

Mb

its/

s


Manc 64bit 66 MHz PCI RedHat 7.1 Kernel 2.4.14 NIC: NetGear

RAL RedHat 7.1 Kernel 2.2.16-3 NIC: Intel pro 1000


23

HI Performance UDP Man - UvA Gigabit I/f

Latency 14.68 mS Slope 0.006 us/byte For 2 PCs expect:

PCI 0.00188

GigE 0.008

PCI 0.00188

Total 0.0118 us/byte n routers extra links ?? No Structure seen:

Throughput 825Mbit/s 1400 bytes Some throughput drop for

packets < 20 us spacing Throughput 725Mbit/s 1200 bytes

Man-UvA Gig 28 Apr 02

y = 0.0061x + 14675

y = 0.001x + 14681

14500

14550

14600

14650

14700

14750

14800

14850

0 500 1000 1500 2000 2500 3000


Late

ncy u

sUDP Man-UvA Gig 28 Apr 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40


Rec

v W

ire

rate

Mb

its/s


Manc Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 NIC: NetGear UvA RedHat 7.1 Kernel 2.2.16-3 ? NIC: NetGear?


24

iperf TCP & UDP throughput MAN-SARA From 20 Oct 01

Iperf TCP throughput Mbit/s

ucl – sara

262144 byte buffer

Forecast

UDPmon throughput Mbit/s

man – sara

300 * 1400 byte frames


25

iperf & Pinger UK-Bologna From 20 Oct 01

Iperf throughput

ucl – Bologna

262144 byte buffer

Forecast in green

PingER rtt (ms)

dl – Bologna

1000 byte packet

Forecast


26Geant Enabled Routing Stable

Iperf ThroughputMbit/sUCL – SARA262144 byte

buffer

UDPmon Loss

ThroughputMbit/sMAN – SARA

iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational


27

Iperf mem-mem vs file copy disk to diskLes Cottrell SLAC

0 400

100

Iperf TCP Mbits/s

File copy dis k-to- dis k

Fast Ethernet

OC3

Disklimited

Over 60Mbits/s iperf >> file copy


28

Don’t Forget Involvement with:

GGF US Grids PPDG, iVDGL UKQCD UKDMC (dark matter) MINOS AstroGRID AccessGRID E-science Centres Optical “Lambda Switching” Projects Collaborations with UKERNA, Dante, Terena …


29

More Information Some URLsPPNCG Home page with Stop Press:

http://ppncg.rl.ac.uk/

and e-mail:[email protected]

DataGrid WP7 Networking: http://www.gridpp.ac.uk/wp7/index.html

DataGrid WP7 EDG Monitoring:http://ccwp7.in2p3.fr/

IEPM PingER home site:http://www-iepm.slac.stanford.edu/

IEPM-BW site:http://www-iepm.slac.stanford.edu/bw

gridpp collaboration meeting may 2002 r. hughes-jones manchester 1 networking in under 30 minutes !...

Documents

u partners

hughesjones manchester

posss ucl osm

ge wan

gbic osm 1oc48

gbic ral osm

traffic statisticaly

gbic mcc osm