Distributed FX software correlation
Distributed FX software correlation
Adam DellerSwinburne University/CSIRO Australia Telescope National
Facility
Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)
Adam DellerSwinburne University/CSIRO Australia Telescope National
Facility
Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)
28/09/200628/09/2006 Torun, PolandTorun, Poland
OutlineOutline
• Software vs hardware correlators• Why software (what it does best,
and the price you pay)• Current applications• The DiFX architecture• Performance and current status• Conclusions
• Software vs hardware correlators• Why software (what it does best,
and the price you pay)• Current applications• The DiFX architecture• Performance and current status• Conclusions
28/09/200628/09/2006 Torun, PolandTorun, Poland
Software vs HardwareSoftware vs Hardware
• “Software correlator”: program running on a supercomputer/cluster
• Software is unclocked, could be faster or slower than real-time
• No channel/integration time restrictions
• Floating pt vs int calculations
• “Software correlator”: program running on a supercomputer/cluster
• Software is unclocked, could be faster or slower than real-time
• No channel/integration time restrictions
• Floating pt vs int calculations
28/09/200628/09/2006 Torun, PolandTorun, Poland
Why softwareWhy software
• Flexibility - can do science that is impossible with hardware correlators
• Rapid (and cheap) development• Add-ons MUCH easier in software• Compatibility and expandability• In general, less approximations due
to the use of floating point
• Flexibility - can do science that is impossible with hardware correlators
• Rapid (and cheap) development• Add-ons MUCH easier in software• Compatibility and expandability• In general, less approximations due
to the use of floating point
28/09/200628/09/2006 Torun, PolandTorun, Poland
The price you payThe price you pay
• Because software & hardware not specifically tuned, less computation per $$ hardware
• Therefore not useful for EVLA/SKA scale correlators
• Running costs (electricity, aircon) may be higher
• Because software & hardware not specifically tuned, less computation per $$ hardware
• Therefore not useful for EVLA/SKA scale correlators
• Running costs (electricity, aircon) may be higher
28/09/200628/09/2006 Torun, PolandTorun, Poland
Current applications/results
Current applications/results
• New science the software correlator has enabled:– Disk-based LBA (greater bandwidth)– Adding global disk-based antennas to
LBA experiments– High time and frequency resolution
allowing wide-field imaging– VERY high frequency resolution for
pulsar scintillation studies
• New science the software correlator has enabled:– Disk-based LBA (greater bandwidth)– Adding global disk-based antennas to
LBA experiments– High time and frequency resolution
allowing wide-field imaging– VERY high frequency resolution for
pulsar scintillation studies
28/09/200628/09/2006 Torun, PolandTorun, Poland
The DiFX architectureThe DiFX architecture
Master NodeMaster Node
Core 1Core 1DataStream 1DataStream 1
DataStream 2DataStream 2
DataStream NDataStream N
Core 2Core 2
Core MCore M
…… ……
Timerange, destinationTimerange, destination
Baseband dataBaseband data
VisibilitiesVisibilities
Source dataSource data
MPI is used for inter-process communicationsMPI is used for inter-process communications
28/09/200628/09/2006 Torun, PolandTorun, Poland
Usage in the LBAUsage in the LBA
• PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas)
• Data is recorded directly to Linux-formatted storage (Apple Xraid)
• Disks shipped to Swinburne• Correlated at the Swinburne
supercomputer (~300 P4 machines)• Ftp fringe tests and 256 Mbps eVLBI
• PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas)
• Data is recorded directly to Linux-formatted storage (Apple Xraid)
• Disks shipped to Swinburne• Correlated at the Swinburne
supercomputer (~300 P4 machines)• Ftp fringe tests and 256 Mbps eVLBI
28/09/200628/09/2006 Torun, PolandTorun, Poland
PerformancePerformance
Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s)
Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s)
Cores required for real-time, 256 Mbps, 256 channels, 1 second integrations
0
50
100
150
200
250
300
0 5 10 15 20
Antennas
Nod
es
28/09/200628/09/2006 Torun, PolandTorun, Poland
Current statusCurrent status
• Verification via correlator comparison with VLBA completed
• Used in production capacity with LBA - complete switch as soon as sufficient disks available
• Further development (graphical frontend, FITS-IDI etc) ongoing
• Exploring potential for geodesy
• Verification via correlator comparison with VLBA completed
• Used in production capacity with LBA - complete switch as soon as sufficient disks available
• Further development (graphical frontend, FITS-IDI etc) ongoing
• Exploring potential for geodesy
28/09/200628/09/2006 Torun, PolandTorun, Poland
ConclusionsConclusions
• Software correlators can be rapidly developed and enable science that cannot be done elsewhere
• Perfect for niche/part-time experiments and feasibility tests
• Used successfully with the LBA• Early science results encouraging!
• Software correlators can be rapidly developed and enable science that cannot be done elsewhere
• Perfect for niche/part-time experiments and feasibility tests
• Used successfully with the LBA• Early science results encouraging!
28/09/200628/09/2006 Torun, PolandTorun, Poland