High Data Rate Transfer for HEP and VLBI

Ralph Spencer, Richard Hughes-Jones and Simon Casey

The University of ManchesterNetwrokshop33 March 2005

HEP: The LHC Detectors

CMS

ATLAS

LHCb~6-8 PetaBytes / year

~108 events/year

Radio Astronomy• The study of celestial objects at <1 mm to >1m

wavelength.• Sensitivity for continuum sources• B=bandwidth, integration time.• High resolution achieved by interferometers.

Some radio emitting X-ray binary stars in our own galaxy:

B/1

GRS 1915+105 MERLIN

SS433MERLIN and European VLBI

Cygnus X-1VLBA

Earth-Rotation Synthesis

Need ~ 12 hours for full synthesis, not necessarily collecting data for all that time.NB Trade-off between B and for sensitivity.

Telescope data correlated in pairs:N(N-1)/2 baselines Merlin u-v coverage

The European VLBI NetworkEVN

• Detailed radio imaging uses antenna networks over 100s-1000s km

• At faintest levels, sky teems with galaxies being formed

• Radio penetrates cosmic dust - see process clearly

• Telescopes in place …• Disk recording at

512Mb/s• real-time connection

allows greater – response– reliability– sensitivity

WesterborkNetherlands

Dedicated

Gbit link

EVN-NREN

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM link

CambridgeUK

MERLIN

MedicinaItaly

Chalmers University

of Technolo

gy, Gothenbu

rg

TorunPoland

Gbit link

eVLBI Milestones • January 2004: Disk buffered eVLBI session:

• Three telescopes at 128Mb/s for first eVLBI image

• On – Wb fringes at 256Mb/s

• April 2004: Three-telescope, real-time eVLBI session.

• Fringes at 64Mb/s• First real-time EVN image - 32Mb/s.

• September 2004: Four telescope real-time eVLBI• Fringes to Torun and Arecibo• First EVN, eVLBI Science session

• January 2005: First “dedicated light-path” eVLBI• ??Gbyte of data from Huygens descent

transferred from Australia to JIVE• Data rate ~450Mb/s

• 20 December 20 2004• connection of JBO to Manchester by 2 x 1 GE• eVLBI tests between Poland Sweden UK and Netherlands at 256 Mb/s

• February 2005• TCP and UDP memory – memory tests at rates up to 450 Mb/s(TCP) and 650 Mb/s (UDP)• Tests showed inconsistencies betweeb Red Hat kernals, rates of 128 Mb/s only obtained on 10 Feb• Haystack (US) – Onsala (Sweden) runs at 256 Mb/s

• 11 March Science demo• JBO telescope winded off, short run on calibrator source done• Test needs to be repeated!

Jodrell BankUK

OnsalaSweden

MedicinaItaly

TorunPoland

EffelsbergGermany

WesterborkNetherlands

Telescope connections

JIVE

1Gb/s

1Gb/s

1Gb/s

155Mb/s

MERLIN

CambridgeUK

2.5Gb/s

1Gb/s

MERLINe

?????

Tests on the Network, Manchester-Dwingeloo: investigation of packet loss

• 4th year MPhys Project at The Univertsity of Manchester Oct-Dec 2003, using campus network and SuperJANET 4 academic network in the UK.• UDPmon used to test throughput and packet loss• NB tune up the end machines – see http://grid.ucl.ac.uk/NFNN.html

Effect on the local traffic:

Gnt5-DwMk5 11Nov03-1472 bytes

0

2

4

6

8

10

12

0 5 10 15 20 25 30 35 40Spacing between frames us

% P

acket

loss

Gnt5-DwMk5

DwMk5-Gnt5

Throughput vs packet spacing Manchester: 2.0G Hz Xeon Dwingeloo: 1.2 GHz PIII Near wire rate, 950 Mbps NB record stands at 6.6 Gbps

SLAC-CERN

Packet loss

CPU Kernel Load sender

CPU Kernel Load receiver

4th Year project Adam Mathews Steve O’Toole

UDP Throughput Manchester-Dwingeloo (Nov 2003)

Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes

0

200

400

600

800

1000

1200

0 5 10 15 20 25 30 35 40Spacing between frames us

Recv W

ire r

ate

Mbits/s

Gnt5-DwMk5

DwMk5-Gnt5

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100

0 5 10 15 20 25 30 35 40Spacing between frames us

% K

erne

l S

ende

r

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100

0 5 10 15 20 25 30 35 40Spacing between frames us

% K

erne

l R

ecei

ver

Packet loss distribution:

( )t

p d

Cumulative distribution

Cumulative distribution of packet loss, each bin is 12 sec wide

Long range effects inthe data?

Poisson

26th January 2005 UDP Tests

Simon Casey (PhD project) Between JBO and JIVE in Dwingeloo, using production networkPeriod of high packet loss (3%):

The Future:

• eVLBI tests in EVN at ~ 1 per month• Network tests at up to 950 Mb/s – statistics of

packet loss• Investigate alternatives to TCP/UDP – tsunami,

VSI-E etc.• ESLEA tests on MB-NG and UKLight• EXPRes EU eVLBI proposal including

distributed processing, JBO-Onsala test link at 4 Gbps, submitted March 19th 2005

• eVLBI will be routine in 2006!!

Processing Nodes

Controller/DataConcentrator

VLBI Correlation: GRID Computation task

Questions ?