new use for an old correlator

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

NEW USE for An old correlatoR

Arpad Szomoru

Joint Institute for VLBI in Europe


Mark IV EVN Correlator

• Developed by international consortium: EVN institutes, MIT

• Officially inaugurated October 1998• Comparable correlators in use at

Haystack Obs., US Naval Obs., MPIfR, JIVE

• Correlator boards in use at WSRT, SMA


Basic specifications

Input:• No. of telescopes (N) 16• Array Observing frequencies: 329 MHz – 22 GHz• Data bandwidth: 128 MHz per polarization (Right and

Left-hand circular), divided into 8 bands. • Channel bandwidths 0.5 MHz to 16 MHz.• Data input rate: 1 Gbps per telescope

Output:• Integration time 1/4s (will become 1/32s with PCInt)• 2048 spectral channels per baseline/band/polarization • Data Output rate 6 MB/s (will become 80 MB/s with PCInt)


Field of View limitation

• Limited by tint

• Time smearing

• Shorter integrations• Enable wide field

surveys• Study μJy sources• Discriminate AGNs

• But, enormous increase of output data volume..

VLBI FoV x 100

6 arcmin [FWHP] Effelsberg beam


EVN MkIV Correlator limits• Integration time

• Cycle time of 0.015s (actually, 1/64th of a second)

• Spectral resolution• 131072 complex lags per readout = 65536 spectral points

per readout• Divided over 32 products leads to 2048 spectral channels

per product

PCINT :• Short for Post Correlator Integrator• Capture the full output of the EVN MkIV

correlator to disk• Need to replace output datapath


The PCInt project

• High speed readout of the correlator was already prepared• Via DSP powered serial port

• Need hardware and software to enable this• Receiving end of serial port• Gbit ethernet for transfer from correlator rack to data collection host• Fast disk subsystem in order to support 160MByte/s (parallel RAID

arrays in a Storage Area Network)

Harro Verkouter


Curre

nt situ

atio

n

CCC

EEE (xk)

DDD (xn)

Switch

FC Switch

Correlator Board (x8)

RaidArray (xm)

RT System

Ethernet Card

C40 COMM

VME High Speed Serial

SBC (x2)

Correlator rack (x4)

PCI

100TX1000FX(x8)

Fibre Channel


Phase

0

CCC

EEE (xk)

DDD (xn)

Switch

FC Switch


RaidArray (xm)

RT System



100TX1000FX(x8)

Fibre Channel

Ethernet Card

C40 COMM

SBC (x2)

PCI


Phase

1

CCC

EEE (xk)

DDD (xn)

Switch

FC Switch


RaidArray (xm)

RT System



100TX1000FX(x8)

Fibre Channel

Ethernet Card

C40 COMM

SBC (x2)

PCI


Phase

2

CCC

EEE (xk)

DDD (xn)

Switch


RaidArray (xn)

RT System



100TX1000FX(x8)

1000TX

Ethernet Card

C40 COMM

SBC (x2)

PCI


Post-processing issues• Huge data volumes

1hour @160MByte/s equals560GBytes of data

• Use a cluster of nodes

• Need automated processing• Use a processing pipeline

Achieved 1/16s sampling, at 24 MB/s data outputUsers seem to be ready for 800 GB

data-sets…


Price recording media ($/GB)


Data Acquisition

•Move from tape to disk recording

•Reliability•Cost•Bandwidth•Efficiency

•e-VLBI: the next step•No consumables•Higher bandwidth•Fast turn-around•ToO support

Disk based recording

e-VLBI using fiber


Why e-VLBI ?

• Reliability – real-time feedback to the telescopes• Logistics – No media management• Sensitivity – sustained data rates >> 1 Gbps possible…

• Rapid science results:

• Geodesy (Earth rotation rate)• Precision spacecraft navigation• Transient phenomena… GRBs, SNe etc.


Why e-VLBI (cont) ? • Target of Opportunity (ToO) capability:

• Dominated by VLBA currently…

• Reliability & Logistics e-VLBI

• Sensitivity e-VLBI

• Rapid science e-VLBI

• Rapid publication e-VLBI

• Optimal observing strategy (obs. freq., calibrators, telescope array)

• LOFAR Transients etc. ToOs may become much more common e-VLBI.


e-VLBI Proof-of-Concept Project

• DANTE/GÉANT Pan-European Network• SURFnet Dutch NREN• GARR Italian NREN • UKERNA UK NREN• PSNC Polish NREN • DFN German NREN• KTHNOC/NORDUnet Nordic NREN • Manchester University Network application software • JIVE EVN Correlator• Westerbork telescope Netherlands• Onsala Space Observatory Sweden• MRO Finland• MPIfR Germany• Jodrell Bank UK• TCfA Poland • CNR IRA Italy


0SISISISI

GRGRGRGR

CZCZCZCZ

IEIEIEIE

LULULULU

EEEEEEEESKSKSKSK

ILILILILLTLTLTLTCYCYCYCY

ESESESES

HUHUHUHUITITITIT

FRFRFRFR

UKUKUKUK

BEBEBEBE

NLNLNLNLDEDEDEDE SESESESE CHCHCHCH

PTPTPTPT

ATATATAT

GEANTGEANTGEANTGEANT

2.5 G 1.2 G 622M

155 M

BGBGBGBGLVLVLVLVRORORORO

HRHRHRHR

PLPLPLPL

45 M34 M

310 M

GÉANT:Access of NRENs to GÉANT

EVN telescope


POC results

• Demonstration of feasibility• Identification of problems• Has led to closer ties with networking community and

generated political interest• Has laid the foundation for the next step forward

(EXPReS):

• I3 proposal to the EC (Communication & Network Development Call)

• Ranked first out of 43 proposals; nearly fully funded to an amount of 3.9 MEuro.


EXPReS – major aims:

• Making e-VLBI an operational astronomical instrument:

• 16 telescopes connected to JIVE at 1 Gbps • Robust real-time e-VLBI operations• Transparent inclusion of e-MERLIN antennas within e-EVN• Target of Opportunity Capability

• Future developments in e-VLBI

• >> 1 Gbps data transfer rates, • extended LOFAR etc. • distributed software correlation.


Expanding the e-VLBI Network


Network testing

•Use existing protocols on currently available hardware

•TCP maximal reliability•Not really required

• Sensitive to congestion•Lot of fine-tuning necessary•And possible

•UDP connectionless• Unaccountable

•Tailor made protocols?•Lambda switching

•Internet weather•Hard to quantify•Hard to pinpoint bottlenecks

0

100

200

300

400

Dw- BoBo- DwDw- Bo*Bo- Dw*

0

200

400

600

800

Dw- Bo

Dw- Tr

Dw- OnM

bps


• February 2005: network transfer test (BWCTL) employing various network monitoring tools involving Jb, Cm, On, Tr, Bologna and JIVE

Network testing (2)

Onsala-Jive

0

200000000

400000000

600000000

800000000

1000000000

1200000000

1 3 5 7 9 11

13

15

17

19

21

23

25

TCP

UDP


First real-time eVLBI Image, 3 telescope observation of gravitational lens, May 2004

First eVLBI science observation, OH masers around IRC10420, Richards et al, Oct 2004

First broadband eVLBI science, detection of theHypernova SN2001em, Paragi et al, astro-ph/0505468


• First open e-EVN Call for Proposals (March 2006)

• First Target of Opportunity Observations (May 2006, Cygnus X-3), Tudose et al. (in prep)…


Current status

• Regular science runs at 128 Mbps with 6 European stations (24 hours)

• Arecibo sometimes participates at 32 Mbps

• Fringes from all European stations at 256 Mbps have been demonstrated, and,

• on single baselines, 512 Mbps


Issues, Developments

Convincing a correlator designed for tape technology to become real-time..

Operational improvements:• Robustness• Reliability• Speed• Ease• Bandwidth• Station feedback


OngoingNew control computers (Solaris AMD servers)• Cut down on (re-)start time• Powerful code development platform• Tightening up of existing code

Other hardware upgrades:• Upgrade existing connectivity from 6*1 Gbps to

16*1 Gbps (lightpaths)• SX optics (fibres + NICs)• Replacement of SU functionality: Mark5A→B:

motherboards, memory, power supplies, serial links.


Conclusions

• e-VLBI is changing the nature of VLBI• Fast response, ToO capability• better quality control, rapid data delivery• New science, higher bandwidths

• Large fields of view• Will allow the study of μJy sources• Or many masers over a large star formation region• Data archive will contain millions of weak sources

• EXPReS will realize an operational e-VLBI network distributed across 1000’s km – a true pathfinder for SKA.

new use for an old correlator

Documents