highly scalable packetised correlators jason manley casper workshop 2009

Highly ScalablePacketised correlators

Jason ManleyCASPER workshop 2009

Contents

• Where we’ve come from

• CASPER

• Correlator design• Dataflow• Engine operations• Scalability• Present and future deployments

In the beginning…

Computing backend

Analogue processor

Control & Monitoring

Big dish

Interferometry…

ATA

Green Bank 300ft

…more recently…

EVLA baseline board(Carlson, 2006)

CASPERCenter for Astronomy Signal Processing and Electronics Research

• Open-source, collaborative environment• Reusable, platform-independent gateware• Modular, upgradeable hardware• Low instrument NRE, shared by the community• Enables rapid development (8 instruments / 2 years)• Industry standard communication protocols• Low Cost

CASPER DBE concept

CASPER DBE concept

Clocking

• Clocks:

• Locally synchronous, globally asynchronous.

• Sampling clock synchronized at F engines.

• X engines each run off independent local clock.

• Synchronized using global 1pps signal at ADCs

• Propagated to X engines using out-of-band signaling on XAUI links.

• Headers labeling and timestamping 10GbE Ethernet packet data.

Correlator Overview

How does it scale?

16 pol, 2k ch, 200MHz

64 pol, 2k ch, 200MHz

256 pol, 16k ch, 512MHz

1024 pol, 16k ch, 1GHz

4096 pol, 2M ch, 1GHz

8192 pol, 2M ch, 1GHz

1

10

100

1000

10000

100000

1000000

F EnginesX Engines

1998 2000 2002 2004 2006 2008 20100

100000

200000

300000

400000

500000

600000

700000

800000

43000100000

200000

330000

760000

Xilinx Virtex Family

• Processing power doubling every two years• V4 = ½ power requirements of V2Pro*

* Manufacturers claim - Xilinx Inc.

Existing systems: PAPER - Green Bank

• 8 dual-pol antennas, 150MHz.• 1st Generation: one BEE2 & four IBOBs.• First deployment of packetized system.• Continuous upgrades.

Existing systems: Northern Cross, Medicina, Italy

• Designed and deployed in under 2 weeks.• 32 single-pol antennas, 32 MHz.• 1st Generation:

• one BEE2• eight IBOBs.

New deployments: KAT-7

• 7 dual-pol antennas, 400MHz.• Pure ROACH.• New control and monitoring interface (KATCP), coarse

delay, high speed output (less than 100ms).• Expected early 2010.

New deployments: GMRT

• 32 dual-pol inputs, 400MHz bandwidth.• Collaboration – shares design with KAT-7.

New deployments: PSA-32/64/128/256

• PAPER in South Africa, staged deployment• Up to 256 antennas, 100MHz • Generation 1.5 (IBOB+ROACH) initially,• Generation 2.5 (ROACH + ROACH-II) eventually.• First field deployment of ROACH system.• Correlator deployment starting in January!

Jason Manleyjason_manley@hotmail.com