Network Tests at CHEPNetwork Tests at CHEP

K. Kwon, D. Han, K. Cho, J.S. Suh, D. SonCenter for High Energy Physics, KNU, Korea

H. ParkSupercomputing Center, KISTI, Korea

Representing the HEP Working Group for ANF/the HEP Data Grid WG

The 3rd International Workshop on HEP Data Grid August 26, 2004, Daegu, Korea

2

IntroductionIntroduction Network Tests using Iperf

Domestic tests International tests (USA, Europe)

Real File Transfer Tests using bbFTP Domestic tests International tests (Europe)

Summary & Future Works

3

HEP Data GridHEP Data Grid Implementation of the Tier-1 Regional Data Center of LHC

Networking Tier0 (CERN)–Tier1 (CHEP) : ~2.5Gbps via TEIN Tier1(CHEP)–Tier1(US and Japan): ~Gbps via APII Tier1(CHEP), Tier2 or 3(inside Korea): 0.155~1 Gbps via

KOREN/KREONET

Computing(1000 CPU Linux clusters)

Data Storage Capability Storage: 1.1 PB Raid Type Disk (Tier1+Tier2) Tape Drive: ~ 3.2 PB HPSS Servers

4

Network at CHEP & Available Research Network at CHEP & Available Research NetworksNetworks

Servers/PCs……

Servers

JapanAPII/H-G

1 G

2.5G

Cisco7606

1 G

Kreonet

CHEP

Clustered PCs ……

APII/KREONET22 * 622 M

USA

TEIN34 M

KORENCERN/Geneva

…100 M

1 G

HSM

TransPAC 2.5G

KNU CC

1 G

Cisco6509

DataTaG 2.5G

Network Test PC

5

Test Tools Test Tools Iperf

A tool for measuring TCP and bandwidth performance. Data sent by default from the client’s memory to the server memory

bbFTP File transfer software that is optimized for large files.

Supports multi-stream transfer and big windows

TCP Reno: Linux 2.4.26 TCP protocol

6

Factors affecting TCP Factors affecting TCP performanceperformance

Window Size

Number of Streams

MTU – We have not tried yet.

Txqueuelen – No gain in performance

SACK – No gain in performance

7

CHEP CHEP KOREN-NOC Iperf TestKOREN-NOC Iperf TestKNU- > KOREN- NOC (Single Stream)

0100200300400500600700800900

1000

256 KB 512 KB 1 MB 2 MB 3 MB 4 MBTCP Window Size

Ach

ieva

ble

ban

dw

idth

in M

bp

s

KNU->KOREN- NOC (5 Parallel TCP Streams)

0

50

100

150

200

250

A B C D ETCP Stream

Achie

va

ble

Ba

nd

wid

th in M

bp

s

RTT: ~2ms

BDP=0.002*1000Mbps=0.25MB

Max throughput: 920Mbps

Throughput of five streams: 916Mbps

KORENKNU

KOREN-NOC1 G

CHEP1 G

GigabitEthernet3/1 -- KOREN

8

Single Stream Tests Single Stream Tests between CHEP between CHEP Caltech Caltech

Duration: 10min each, 10min interval, over KOREN-TransPAC path (1Gbps), 20MB window, TCP(Linux 2.4.26)

KNU- > Caltech(June 23)

0

50

100

150

200

1 2 3 4 5No. of Trial

Ban

dwid

th(M

bps)

KNU Genkai LA CaltechBusan

1 G 2.5G1 G

KOREN H-G transPAC CalREN2

Tokyo1G 1 G

RTT: ~130ms

BDP: 0.13*1000Mbps=16MB

Max throughput: 146Mbps

9

Multi-Stream Tests between Multi-Stream Tests between CHEP CHEP Caltech Caltech

CHEP- >Caltech (June 23)

0

200

400

600

800

1000

10 20 30 40 50 60 70 80 90 100No. of Streams

Band

wid

th(M

bps)

Duration: 10min each over KOREN-TransPAC path (1Gbps),

(Stream* Window ) <=100MB

APII/Genkai Link (APII-Juniper ge-0/1/0.1)

TransPAC LA link (TPR2 so-2/0/0.0)

Max throughput: 783Mbps

10

Single Stream Tests between Single Stream Tests between CHEP CHEP CERN CERN

KNU- >CERN(Single Stream)

020406080

100120

1 2 3 4 5No. of Trials

Thro

ughp

ut(M

bps)

KNU Genkai LA ChicagoBusan

1 G 2.5G1 G

KOREN H-G transPAC Abilene

Tokyo1G 1 G

CERN2.5 G

DataTag

Duration: 10min each, 10min interval, over KOREN-TransPAC path

40MB window, TCP(Linux 2.4.26)

RTT: ~370ms

BDP: 0.37*1000Mbps=46MB

Max throughput: 99Mbs

11

Multi-Stream Tests between Multi-Stream Tests between CHEP CHEP CERN CERN

CHEP- >CERN(Multistream)

0100200300400500600700800

10 20 30 40 50 60 70 80 90 100No. of Streams

Thro

ughp

ut(M

bps)

APII/Genkai Link (APII-Juniper ge-0/1/0.1)

TransPAC LA link (TPR2 so-2/0/0.0)

Duration: 10min each over KOREN-TransPAC path (1Gbps),

(Stream* Window ) <=100MB

Max throughput:714Mbps

12

Other TCP StacksOther TCP Stacks

CHEP->CERN(Single Stream)

050

100150200250300350

1 2 3 4 5No. of Trials

Throu

ghpu

t(Mbp

s) HS-TCPTCP RenoFAST

Setup for HS-TCP:net.ipv4.tcp_rmem= 4096 87380 67108864net.ipv4.tcp_wmem= 4096 87380 67108864net.ipv4.tcp_mem= 8388608 8388608 67108864txqueuelen =1000

Setup for FAST TCP:net.ipv4.tcp_rmem= 4096 33554422 134217728net.ipv4.tcp_wmem= 4096 33554422 134217728net.ipv4.tcp_mem= 4096 33554422 134217728txqueuelen =1000

CHEP- >Caltech(Single Stream)

050

100150200250300350400450

1 2 3 4 5No. of Trials

Thro

ughp

ut (

Mbp

s)

HS- TCPTCP RenoFAST

13

Real File Transfer between Real File Transfer between KNU & KOREN over Linux TCP KNU & KOREN over Linux TCP [root@cluster90 bbftpc]# ./bbftp -V -e 'setrecvwinsize 1024; setsendwinsize 1024; put ams' -u root 203.255.252.26

Password:

>> USER root PASS

<< bbftpd version 3.0.2 : OK

>> COMMAND : setremotecos 0

<< OK : COS set

>> COMMAND : setrecvwinsize 1024

<< OK

>> COMMAND : setsendwinsize 1024

<< OK

>> COMMAND : put ams ams

<< OK

1024000000 bytes send in 19.8 secs (5.06e+04 Kbytes/sec or 395 Mbits/s)

14

I/O test run rulesI/O test run rules The maximum file size to be greater than the

total physical memory to get accurate results (Iozone file system benchmark)

Perform 40X physical RAM size worth of IO to minimize the percentage of error due to IO being read out of cache(3ware white paper)

15

Real File Transfer (100GB) between Real File Transfer (100GB) between KNU & KORENKNU & KOREN

KOREN(2.5G)

AMD Opteron Dual, Tuned RAID 0, (Read 197MB/s with Iozone, Write 178MB/s)

Xeon 2GHz Dual, ATA DISK Drive at KOREN-NOC, Daejon

1G 1G

Time Taken: 1 hour 20 min 58

Average throughput: 164Mbps

16

Real File Transfer(1TB) between Real File Transfer(1TB) between KNU & KORENKNU & KOREN

KOREN

KNU File Server

(Tuned RAID 0, Read 197MB/s with Iozone, Write 178MB/s)

machines at KOREN-NOC in

Daejon

1G

1G

1G

1G

1G

1G

200GB each

A: 3 hr. 13 min. 20

B: 3 hr. 13 min. 31

C: 3 hr. 14 min. 36

D: 3 hr. 14 min. 46

E: 3 hr. 11 min. 22

E

D

C

A

B

Throughput: 701Mbps

17

File Transfer (100GB)with LustreFile Transfer (100GB)with Lustre

Lustre(Linux + Cluster): Distributed file system for large clusters

OST

OST

OST

OST

OSTClient

MDS

GigE

100GB

1G

KNU File Server

Time Take: 52min 55

Throughput: 251Mbps

18

Real File Transfer between KNU Real File Transfer between KNU & CERN over HS-TCP& CERN over HS-TCP

[kihwan@w01gva bbftpc]$ ./bbftp -V -e 'setrecvwinsize 41024; setsendwinsize 41024; cd /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd; get ams' -u root cluster90.knu.ac.krPassword: >> USER root PASS<< bbftpd version 3.0.2 : OK>> COMMAND : setremotecos 0<< OK : COS set>> COMMAND : setrecvwinsize 41024<< OK>> COMMAND : setsendwinsize 41024<< OK>> COMMAND : cd /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd<< OK : Current remote directory is /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd>> COMMAND : get ams ams<< OK1024000000 bytes got in 47.7 secs (2.1e+04 Kbytes/sec or 164 Mbits/s)

SummarySummary High Bandwidth Network is essential for HEP Data Grid

Domestic Links Only Window size need to be adjusted to fully utilize available

bandwidth Real file transfer shows that speed is limited by physical I/O,

rather than network

International Links Single stream: ~100Mbps Parallel streams needed to achieve significant throughput Other TCP Stacks may help improve performance Further tests and investigations should be done

20

Future WorksFuture Works

Jumbo Frame (9000Byte MTU)

File transfer using RAID disks between KNU and CERN

Tests over Lambda Networks

21

Terabyte Transfer TestTerabyte Transfer Test

Equipments (4TB Raid, 2 Network Test machines)

CHEP File servers

KREONET

IPv6 (with Lambda)

2.5 GbpsKISTI