e-VLBI Development at Haystack Observatory

Alan WhitneyChet Ruszczyk

MIT Haystack Observatory

10 Jan 2006IVS General Meeting

Concepion, Chile

Scientific Advantages of e-VLBI

• Bandwidth growth potential for higher sensitivity– VLBI sensitivity (SNR) proportional to square root of Bandwidth

resulting in a large increase in number of observable objects(only alternative is bigger antennas – hugely expensive)

– e-VLBI bandwidth potential growth far exceeds recording capability(practical recordable data rate limited to ~1 Gbps)

– e-VLBI is a key technology to realize goals of VLBI2010 • Rapid processing turnaround

– Astronomy• Ability to study transient phenomena with feedback to steer observations

– Geodesy• Higher-precision measurements for geophysical investigations

• Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

Highlights of recent e-VLBI developments

• August 2004:– Network link to Haystack upgraded to 2.5 Gbps

– Real-time fringes at 128 Mbps, Westford and GGAO antennas, Haystack Correlator

• February 2005– Real-time fringes Westford-Onsala at 256 Mbps– Used optically-switched light paths over part of route

• Starting April 2005– Start routine e-VLBI transfers from Tsukuba and Kashima

• Starting ~June 2005– Automated regular e-VLBI UT1 Intensive data transfers from Wettzell

• September 2005– CONT05 data from Tsukuba transferred to Haystack via e-VLBI

• Fall 2005– Effort initiated to connect NyAlesund to Haystack

• November 2005– Global real-time e-VLBI demos at 512 Mbps

Challenges

• Network bottlenecks well below advertised rates• Performance of transport protocols

– untuned TCP stacks, fundamental limits of regular TCP• Throughput limitations of COTS hardware

– Disk-I/O - Network• Complexity of e-VLBI experiments

– e-VLBI experiments currently require significant network expertise to conduct• Time-varying nature of network• Standard formats for transfer of data and control information between

different VLBI systems• ‘Last-mile’ connectivity to telescopes

– Most telescopes are deliberately placed in remote areas– Extensive initiatives in Europe and Japan to connect;

U.S. is lagging

Current Projects at Haystack Observatory

• Network interfacing equipment for e-VLBI– Mark 5 VLBI data system

• Standardization (VSI-E)

• Intelligent Applications– Automation of e-VLBI transfers an ongoing process

– Development of optimization-based algorithms for intelligent applications ongoing (EGAE)

– Intelligent optically-switched networks (DRAGON)

• e-VLBI test experiments

• Production e-VLBI– Put e-VLBI into routine use – progressing well in limited venues

• Support e-VLBI development for VLBI2010 initiative

Mark 5 and e-VLBI

• “Triangle of connectivity” allows flexibility in e-VLBI operations• Achieving full 1024Mbps e-VLBI is a challenge with current generation

motherboards and NIC cards• In lab, Mark 5A back-to-back sustained transfers have been achieved at ~1200Mbps

– High-end motherboard; careful choice of NIC cards– Dual aggregated GigE links– Careful tuning

• Robust e-VLBI at 512 Mbps is realizable over real networks• New generation of motherboards and 10GigE NIC cards should allow routine e-VLBI

at 1024Mbps

VLBI Data Port/FPDP bus

Disc Array

64-bit/66MHzPCI Bus

Triangle of connectivity

VSI-E

• Goals:– Efficient transport mechanism– Standard protocols– Internet-friendly transport– Scalable Implementation– Ability to transport individual data-channel streams as individual packet

streams– Ability to make use of multicasting to transport data and/or control information

in an efficient manner• could be used in the future for support of distributed correlation

VSI Model

RTP Architecture

VSI-E Status

• Beta version of VSI-E is being tested in transfers from Kashima to Haystack (has been a bit delayed by network problems);expect result soon

• Plan to submit VSI-E protocol for approval by Internet Engineering Task Force (IETF) as international standard after agreement within e-VLBI community and successful demonstrations

New Application-Layer Protocols for e-VLBI

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed which are tailored to e-VLBI characteristics; for example:– Can tolerate some loss of data (perhaps 1% or so) in many cases

– Can tolerate delay in transmission of data in many cases

• EGAE allows user to specify a profile of experiment requirements, which are then translated into network requirements and strategies

• ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy developed at Haystack Observatory under NSF grant– Now used in some routine e-VLBI transfers

EGAE Progress

• ‘Production’ e-VLBI facility has been established at Haystack to support routine e-VLBI transfers

• EGAE is now supporting routine non-real-time e-VLBI data transfers from Tsukuba

• EGAE will soon be used for routine e-VLBI transfers from Wettzell and NyAlesund

Haystack

Westford

NASA/GSFCUSNO

(correlator)

(correlator)

(Early demo – supported by DARPA)

Bossnet 1 Gbps e-VLBI demonstration experiment

Oct 2002(sustained rate of ~768 Mbps achieved non-real-time)

Initial experiment

Future

Real-time e-VLBI SC2004 Demo

Bossnet

DRAGON

Haystack

Westford

Goddard

GGAO

Pittsburgh Convention Center

128 Mbps

128 Mbps

Real-time e-VLBI SC05 DemoNov 2005

Real-time transmission and processing of data from antennas in Westford, MA, Greenbelt, MD, and Onsala, Sweden at 512 Mbps/antenna

All except Kashima equipped with Mark 5 data systems; Kashima uses Japanese K5, included via VSI-E

Correlation results displayed inreal-time at SC05 meeting

Progress towards routine e-VLBI

• April 2005– Start routine e-VLBI transfer from Kashima and Tsukuba (>200Mbps)

• Starting ~June 2005– Automated regular e-VLBI UT1 Intensive data transfers from Wettzell

to ISI-E (disks hand-carried to USNO for correlation)– A few start-up problems, but now operating fairly smoothly

• Spring 2005– Commitment to connect Hobart via optical fiber (schedule unknown)

• September 2005– All CONT05 data from Tsukuba transferred to Haystack via e-VLBI

(~15 TB!)– Also – all Syowa data transferred via e-VLBI from Japan to Haystack

• November 2005– Project initiated to connect NyAlesund to Haystack through

NASA/GSFC at up to 100Mbps– Expect routine e-VLBI from NyAlesund within a few months

• December 2006– Funds secured to connect Forteleza at 2.5 Gbps

Scorecard of Antenna/Correlator Connectivity(geodetic sites shown in red)

• JIVE Correlator (6 x 1 Gbps)• Haystack (2.5 Gbps)• Westford, MA (10 Gbps to Haystack; 1 Gbps to outside world)• Kashima, Japan (2.5 Gbps)• Usuda, Japan (2.5 Gbps)• Nobeyama, Japan (2.5 Gbps)• Koganei, Japan (2.5 Gbps)• Tsukuba, Japan (2.5 Gbps)• GGAO, MD (1 Gbps)• Onsala, Sweden (1 Gbps)• Torun, Poland (1 Gbps)• Westerbork, The Netherlands (1 Gbps)• Medicina, Italy (1 Gbps)• Jodrell Bank (1 Gbps)• Arecibo, PR (155 Mbps)• Wettzell, Germany (~30 Mbps)• Kokee Park, HA (nominally ~30 Mbps, but currently disconnected)• TIGO (~2 Mbps)

In progress• Hobart – agreement reached to install high-speed fiber• NyAlesund – work in progress to provide 100Mbps link for e-VLBI data

to Haystack Observatory• Forteleza – funds secured for fiber connection at 2.5Gbps; early 2006• Metsahovi – 1Gbps in 2006

International e-VLBI workshops

• Held every year since 2002, rotating between U.S., Europe, Japan and Australia

• Successful e-VLBI workshop held in Sydney, Australia in July 2005

– Hosted by CSIRO/ATNF

– 60 attendees from 9 countries

– Many up-to-date developments in e-VLBI and network research

– e-VLBI Technical Working Group re-formed

– Many thanks to all for a fine workshop

• Next e-VLBI workshop will be held at Haystack, probably in mid-Oct 2006

– YOU ARE INVITED!