evn vlbi verkouter@jive.eu tow may 2017 · n17c1 jive 6cm nme 512 0.69 l+r 6cm nme + ftp-ft. cl17c1...

EVN e-VLBI TOW May 2017verkouter@jive.eu

Intarwebs

Harro Verkouter

e-transfer?

Harro Verkouter

e-transfer?

Harro Verkouter

Real-time VLBI!

• anything/anyone **** up?

• anything/anyone not keeping up?

you lose science data!

Why on earth e-VLBI?

‘Old style’ VLBI ...

Dec Jan Dec Jan

∆t ≫several months!

1. Address time scaleget data to scientist faster

Dec Jan Dec Jan

OHNOES!

2. Reliabilitydetect + fix errors before science run

Sensitivity

Sensitivity ∼ 1 ∕σT

√(∆ν ⋅ ∆t)σT =

σT noise∆ν bandwidth

∆t integration time

√(∆ν ⋅ ∆t)σT =

∆ν☝︎ ⇒σT☟

Recorded data rate ∆ Sensitivity (wrt 1Gbps)1 Gbps + 0%2 Gbps + 41.4213562...%4 Gbps + 100%

`whole’ EVN equipped with• DBBC2• FiLa10G

⇒ 4Gbps capable

F17L1 JIVE 18cm FTP-FT 512 0.23 L+R 18cm FTP Fringe TestEP103A Paragi FRB121102 1024 5.99 L+R 18cmCL17L1 Gunn 18cm FS CAL ---- 0.00 L+R 18cm Amplitude Calibration EM127A Maan B0525+21 1024 3.26 L+R 21cmEM127B Maan B2045-16 1024 2.53 L+R 21cmEK036C Kirsten Crab Nebula 1024 0.92 L+R 18cm 3rd epoch N17L1 JIVE 18cm NME 512 0.69 L+R 18cm NME + FTP-FTGV022A Varenius Arp 220 2048 11.06 L+R 18cm +VLBA+Y27EP102 Paragi J0959+1259 1024 4.61 L+R 18cm +eMERLINN17C1 JIVE 6cm NME 512 0.69 L+R 6cm NME + FTP-FTCL17C1 Gunn 6cm FS CAL ---- 0.00 L+R 6cm Amplitude CalibrationEP103B Paragi FRB121102 2048 11.06 L+R -EL056 Landoni SDSS 0040-0915 2048 5.53 L+R -N17P1 JIVE 90cm NME 256 0.35 L+R 90cm NME + FTP-FTCL17P1 Gunn 90cm FS CAL ---- 0.00 L+R 90cm Amplitude CalibrationGK049C Kirsten B1133+16 256 2.88 L+R +GBT+Y27 B1133+16GK049D Kirsten B1737+13 256 2.88 L+R +GBT+Y27 B1737+13F17K1 JIVE 1.3cm FTP-FT 1024 0.46 L+R 1.3cm FTP Fringe TestGG081C Gomez AGN Jets 1024 6.91 L+R 1.3cm EVN+RA OJ287EC057B Cutini S51803+78 128 0.35 L+R 2nd epochN17K1 JIVE 1.3cm NME 1024 0.69 L+R 1.3cm NME + FTP-FTGG081D Gomez AGN Jets 1024 0.46 L+R +RA 0823+033GG081E Gomez AGN Jets 1024 0.69 L+R +RA OJ287GG081F Gomez AGN Jets 1024 0.69 L+R +RA+VLBA (HnScNl) OJ287GG081G Gomez AGN Jets 1024 0.46 L+R VLBA ONLY+RA 0823+033 EB060A Bach Cygnus A 1024 4.61 L+R -GG081H Gomez AGN Jets 1024 0.69 L+R VLBA ONLY +RA OJ287CL17K1 Gunn 1.3cm FS CAL ---- 0.00 L+R 1.3cm Amplitude CalibrationGG081I Gomez AGN Jets 1024 0.69 L+R +RA OJ287GG081J Gomez AGN Jets 1024 0.69 L+R +RA+VLBA (HnScNl) OJ287F17Q1 JIVE 0.7cm FTP-FT 1024 0.46 L+R 0.7cm FTP Fringe TestGK050 Koyama Mrk 501 1024 6.91 L+R 0.7cm +VLBAEP101A Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101B Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2EP101C Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101D Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2GG081L Gomez AGN Jets 1024 0.69 L+R 1.3cm VLBA ONLY +RA OJ287GA032D Agudo NRAO 150 1024 5.33 L+R 0.7cm +VLBAEP101E Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101F Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2N17Q1 JIVE 0.7cm NME 1024 1.38 L+R 0.7cm NME + FTP-FTCL17Q1 Gunn 0.7cm FS CAL ---- 0.00 L+R 0.7cm Amplitude CalibrationEP101G Pilhstrom Q-band Calibrators 1024 0.46 L+R 0.7cm field 1EP101H Pilhstrom Q-band Calibrators 1024 0.92 L+R 0.7cm field 2

ession I/2017

EVN Sensitivity

> 1Gbps observations sparsesolely

due to recording media shortage!

3. Sensitivity/capacitynot limited by recording media any more

4. FUN!!playing with novel technologies!

CORIntarwebs

Intarwebs

Thus: e-VLBI necessities1. get your `telescope’ connected

• fix “last mile problem”• 2017: sometimes still prohibitively expensive!

• may need sucking up to Research Network Provider• some charge commercial rates:

• ~ few 1000 currency units / time unit / bandwidth• 2017: sometimes still prohibitively expensive!

• sometimes even political issues i.s.o. money

2. switch to Mark5 recorder or newer

Any of these fail, you can forget it

CORIntarwebs

Simplified network diagram!

Intarwebs

control s/w

Mark5 system

Anatomy of Mark5

PCI bus

I/O board

StreamstorCard

bank a bank b

control s/w

Mark5 system

PCI bus

I/O board

StreamstorCard

bank a bank b

record = on / off

control s/w

Mark5 system

in2net = on / off

PCI bus

I/O board

StreamstorCard

bank a bank b

• TCP• TCP BIC• TCP Reno• ...

Started off with TCP

Long fat pipes: large BW x delay producte.g. RTT Netherlands - Australia ~ 270ms⇒ unfavourable for TCP

“let’s try udp”

we were in for a few surprises ...

• connections were dedicated light paths• did not have to be very polite to other users• we can tolerate loss of ~ few % easily

• our data is pure noise!

Current:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);

The n00b does UDP ...

Current:

• packets don’t even leave the Mark5! • udp has no rate control/feedback (handled by O/S for tcp)

• kernel: no obligation to actually deliver the packet(s) ...• sender overwrites own send buffer ...• have to invent ingestion rate control

The n00b does UDP ...

Use Linux nanosleep() system call:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);nanosleep( delta_t );

For 1Gbps/MTU1500 delta_t(δt) is ~12 microseconds.Linux not real-time O/S:

threads only scheduled every 10ms (100Hz)!

n00b reads manpage

δt (~10ms)!

Use Linux nanosleep() system call:

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);nanosleep( delta_t );

n00b reads manpage

Use (pseudo) calibrated busy-loop:

int N = delta_t * counts_per_microsecond;

for( pkt=0; pkt<npkt; pkt++) {write(socket, data, 1500);for(int dummy=0; dummy<N; dummy++);

n00b thinks he smart

δt (random)

Use wallclock time; busy wait until time for next packet:

for( pkt=0; pkt<npkt; pkt++) {// remember time of sendingtime_sent = gettimeofday();// send packetwrite(socket, data, 1500);// wait for wallclock time to pass ...time_next = time_sent + delta_t;while( gettimeofday() < time_next ) {}

n00b don’t give up

n00b now know how!

0 1 2 3 4 5 6 97 8

So far only sender side!

UDP = no guarantees:packet losspacket reordering

Sender sends in sequence:

4 1 3 5 6 7 9 138 12Receiver gets this:

0 1 2 3 4 5 6 97 8

Put back in order

4 1 3 5 6 7 9 138 12

Sender sends in sequence:

Receiver gets this:

receiver must do this in real-time

Put back in order• 3 generations of algorithm, up to 2048Mbps

• initializes buffer by overwriting (memcpy)• lots of statistics:

• reordering amount (RFC4737-ish)• loss statistics etc.

• 4th generation good up to 8192Mbps test• do not initialize• flag positions with data• only overwrite locations w/o data

• memory touched exactly once• still all statistics

Intarwebs

CORIntarwebs

bsNeed Nsta ⋅ x Gbps

bs1 x 40Gbps Haystack

bs3 x 10Gbps JIVE

bs Again: “last mile problem”In 2017:still not solved ∀correlators

FPGA: delay, cornerturn

Custom chip: correlate

FPGA: delay, cornerturn

Custom chip: correlate

Custom chip: correlateextremely synchronous ...• no buffering• no handling of network jitter

FPGA: delay, cornerturnextremely synchronous ...• lost sync if too few valid time stamps found

• e.g. due to packet loss ...• correlation breaks badly!

• also no buffering

CPU: delay, cornerturn, correlate

CPU: delay, cornerturn, correlate extremely asynchronous!• no guaranteed throughput, may not keep up• need ~ few 100 compute nodes for 16 Gbps

• CPUs inefficient for 2-bit data

CPU: delay, cornerturn, correlate • per packet overhead is a concern• takes to handle bad/missing data

• node waiting for data that will never arrive• look valid but is invalid: hard to detect!

CPU: delay, cornerturn, correlate • JIVE’s SFXC software correlator works!• Cluster internal network (architecture and

throughput) is important• Investe in enough compute nodes

Something completely

different

CORIntarwebs

Data flow

Correlator Control GUI

CORIntarwebs

Control flow!

Who does what in e-VLBIField System Correlator control

time sync formatter (which channels)

frequency setup start/stop transfer

antenna (re)configure network

recorder (re)configure destination

Correlator must have direct access to sender!

JIVEThe NetYour station

ccs.jive.eu

fs.yourstation.tld

TCP/IP controlFirewall issues

Monitoring

Real-time VLBI!

• anything/anyone **** up?

• anything/anyone not keeping up?

you lose science data!

if( problem )figure_out_why( time(NULL)* );

time(NULL) = ‘now’

http://services.jive.nl/sfxc/fringe.html

Real-time fringeplot on the web

Daunting?Yes.

Real-time VLBI requires:a lot of knowledgeabout a lot of subjectsby a lot of peoplesupported by a lot of tools

Real-time VLBI in EVN

Organized as monthly 24hr sessionsFuture e-VLBI Observing Sessions (real-time correlation)

-------------------------------------------------------2016 Dec 06 - Dec 07 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Jan 17 - Jan 18 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Feb 14 - Feb 15 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Mar 23 - Mar 24 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Apr 11 - Apr 12 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 May 09 - May 10 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm2017 Jun 20 - Jun 21 (start at 13 UTC) 18/21 cm, 6 cm, 5 cm or 1.3 cm

http://www.evlbi.org/evlbi/e-vlbi_status.html

ToO ~ scheduled on few days notice

4hr of test/setup

1. clocksearch2. possible station changes

new network h/wnetwork setup, IPv4uses diff. equip from scheduled

3. AOB

Danke schön!

evn vlbi verkouter@jive.eu tow may 2017 · n17c1 jive 6cm nme 512 0.69 l+r 6cm nme + ftp-ft. cl17c1...

Documents

nme presentation

6cm 1) find the area of this circle: 6cm. 1) find the area...

pre-lesson assessment: glasses · glasses glass 2 6cm 6cm...

%6cm 4 1 %20cm ias— fj6cm 675' Ë3cm%6cm 4 1 %20cm ias—...

ニッケルおよび高ニッケル合金の溶接 ·...

pengantar nme

challenge 5cm 6cm 12 cm 7cm solution: sa = (b x h) + (a x l)...

nme magazine

blossoms nme

nme & kerrang

stichting nme mundial › wp-content › uploads › 2016...

6cm×6cmのイメージ画像品番：rose corsage -...

responsi nme reaksi

the rise and rise of 6cm eme -...

transamination reactivity of ti(nme and zr(nme model...

analysing ‘nme’

y'] / — http ://urokonohandmade. seesaa. net/ 6cm 1...

nme history

adoquin 6cm

nme contents