cross- correlators for radio astronomy

Brent Carlson, 2012 NRAO Synthesis Imaging Summer School

Cross-correlators for Radio Astronomy

ATP-NSI

Brent Carlson, NRAO Synthesis Imaging Summer School29 May 2012

Brent Carlson, 2012 NRAO Synthesis Imaging Summer School 2

Outline

• What is the purpose of the correlator?• Simplified signal flow• Basic correlator architectures

- XF, FX, hybrid• Technology

- How do the electronics “work”• The development process• JVLA WIDAR correlator• Now and the future


Purpose

• To calculate the integrated cross-power response for each pair of antennas “X” and “Y” in the array over some integration time “T”.

T

dttytxT

XY0

)()(1

Statistically independent signals integrate down, whereas the common radio source signal establishes some power level and the uncertainty in that level integrates down.


Purpose

• These are the visibilities—spatial Fourier components—for each baseline B in the u-v plane that are used to build the image.

• The fun begins:- As N increases, number of pairs of antennas goes as ~N2 / 2.

(e.g. JVLA has 27 antennas…378 baselines (including “auto” correlations).

- As bandwidth and number of (spectral) channels increases, the speed of the electronics increases along with power and cost.


Purpose

• Analog correlators, operating on analog (continuous time) voltage signals have been built and do work…

• But nowadays the analog signal is quantized—in time and amplitude—as soon as possible for stability and to take advantage of “cheap” high-speed digital electronics.- Once the signal is digitized there are no more

unknown/unquantifiable effects (well, unless something broke…)


Simplified signal flow

• STEP #1:- Receive and amplify the signal from the antenna.

• The LNA adds significant noise to the weak signal, and must be optimized to optimize sensitivity of the telescope.

• Additive noise stability (or calibrated as such) here is important for stable correlator output.



• What does the signal “look” like?- Time domain (analog):

FL1n

n

Voltage

Time



• STEP #2:- down-convert (mix) and filter the signal…ready for

digital sampling.• Gain fluctuations here not so critical as long as additive noise is stable.

•It is the ratio of radio source signal to system noise that is important.



• STEP #3:- quantize (digitize/sample) the signal.

• One of the most critical+difficult steps in the processing chain because it is where the quiet analog world meets the noisy digital world.

• Speed is important to maximize bandwidth.


“Flash” A-to-D converter diagram.



• What does the signal look like?- Time domain (digital):

FL1n

n

sample# (time)

level/binary code

0000

1111



• Sampling:- Nyquist sampling theorem: must sample at least 2X the

highest frequency content to obtain all information about the signal. If less, leads to “aliasing” (confusion).

- With noise input:- 1-bit results in ~35% sensitivity loss.- 2-bit results in ~12% sensitivity loss.- 3-bit: 3.5% -- JVLA wideband samplers- 4-bit: 1.5% -- JVLA correlator “internal samplers”



• Sampling:- adding more bits/sample produces diminishing

sensitivity returns for noise input and integrated output.- When narrowband interference is present, need more

bits so as not to contaminate the spectrum with saturated sampler-generated harmonics. Get ~6 dB per bit dynamic range for a pure tone. dB=10log(x); if x is a power value.

- For real-time signals (music/video) need lots of bits to accurately represent the real-time waveform (e.g. CD ~16-bit sampling=216 = 65,536 levels)



• STEP #4: - Correct for antenna-dependent wavefront delay.

• Two phases:

1) pure digital delay to +/-0.5 samples. Cheapest is to use memory. Get up to +/-90 deg phase changes at the upper edge of the band…severe decorrelation, therefore need:

2) sub-sample delay to << +/- 0.5 samples. Various methods, sometimes analog, often digital…more later.



• STEP #5:- Cross-correlate and accumulate.

• Must also correct for “fringe phase” due to the fact that wavefront delay compensation occurs at a different frequency (baseband) than where it originally occurred (at RF in free space).

• Various correlator methods to be discussed later…



• What does the signal look like?- Frequency domain (10e6 samples integrated):

Amplitude vs Frequency (bin)

Frequency (bin)

3

0

FBFq 2

N2

0 q



• What does the signal look like?- Frequency domain…add some lines:


Frequency (bin)

FBFq 2

q


Correlator architectures

• There are two basic methods for correlation:- “XF”: Cross-correlate in the time (tau) domain, then

Fourier transform (after integration) to the frequency domain. a.k.a. “lag correlator”

- “FX”: Fourier transform in the (real) time domain, then multiply and integrate in the frequency domain.

“Convolution in one domain is multiplication in the other domain”



• Hybrid:- Combination of the two. JVLA WIDAR does this as

does the ALMA correlator.- Coarse filter into sub-bands, XF each sub-band.- More details later.



• XF:- traditionally simpler to understand+implement—especially for

1-bit or 2-bit correlators (e.g. 1-bit correlator multiplier is XOR gate). Important in “earlier days” because of speed and logic availability.

- Data Nant2 distribution in correlator still uses the same few bits.

- O(Nchan x sample rate) multiplies/sec…but, very simple operations (multiply-accumulate).


Fsinc sx( )

sx

CMAM1

CMAM0

CMAM3

CMAM2

CMAM5

CMAM4

CMAM7

CMAM6

X

Y


Frequency (bin)

3

0

FBFq 2

N2

0 q

x(n)^

y(n)^


0 16 32 48 64 80 96 112 1280

0.2

0.4

0.6

0.80.69064626

0

CL2q

N0 q


0 16 32 48 64 80 96 112 1280

0.1

0.20.19748877

0

CL2q

N0 q


0 16 32 48 64 80 96 112 1280

0.05

0.1

0.15

0.20.16159253

0

CL2q

N0 q


0 16 32 48 64 80 96 112 1280

0.05

0.1

0.15

0.20.16023643

0

CL2q

N0 q


0 16 32 48 64 80 96 112 1280

0.05

0.1

0.15

0.20.16525955

0

CL2q

N0 q



• FX:- More complex, many-bit operations (FFT). (Has been)

more difficult to implement.- O(log Nchan) x sample rate multiplies/sec…much more

efficient…in principle.- Problems:

1. Have word-width expansion after FFT: (has been) 1 or 2-bit in, many bits out.

2. How to window the real-time data before FFT?


Real-timeFFT

Real-timeFFT

x(n)^

y(n)^

X (m)f

Y (m)f

MemoryOne complex MAC

Memory: one accumulator for each frequency

channelAmplitude vs Frequency (bin)

Frequency (bin)

3

0

FBFq 2

N2

0 q

each frequency channel is a time series of samples each at a sample rate of

N/F



• FX problems:- Word-width expansion:

- Keep full word width: large Nant2 data distribution problem in

correlator.- Reduced word width quasi-floating point. Original VLBA correlator

did this.- Re-quantize to ~4 bits. Not much sensitivity loss. With strong

interference just toss the bad channel. With strong source lines…measure power in each channel before re-quantization, correct levels afterwards.

- Keep reasonable # bits to accurately represent the dynamic range of signals possible…no pre-requantizer power measurement required.



• FX problems:- How to window the real-time data before the FFT?

- Don’t…use “box-car” with no overlap…large ringing/sidelobes (sin(x)/x = “sinc” function) but if signal is just noise it is ok. Reduces SNR slightly since all of the data isn’t being correlated.

- Triangular 50% overlap…reduces ringing/sidelobes (sinc2), no SNR loss since all data is used. Orig VLBA correlator did this.

- Poly-phase FIR filter…excellent!



• Poly-phase FX:- Put poly-phase FIR (Finite Impulse Response) filter in front of

the FFT.- Acts like a window/taper function that is longer than the FFT…

achieves superior channel-to-channel isolation and spectral dynamic range. ~10 taps per frequency channel is required.- This performance cannot fundamentally be obtained with an

XF correlator: window function limited to size of FFT, and causes broadening of spectral features.

- ~2X the amount of logic to implement over just the FFT.- Now not such a problem with modern high-performance

programmable devices (FPGAs).


“Ringing” caused by truncated FFT.a.k.a. “Gibb’s phenomenon.

Caused by convolving the true spectrum with a sinc() function.

Ringing greatly reduced by tapered windowing before FFT but lines are broadened.


Real-timeFFT

Real-timeFFT

x(n)^

y(n)^

X (n)f

Y (n)f

MemoryOne complex MAC

Memory: one accumulator for each frequency

channelAmplitude vs Frequency (bin)

Frequency (bin)

3

0

FBFq 2

N2

0 q

Pol

y-ph

ase

FIR

Pol

y-ph

ase

FIR


C0

C1

C2

C3

C4

C5

C6

x(n)^

x(n) * H(n) X(f) x H(f)^

Fsinc sx( )

sx

H(n)=”impulse response”

C0 C1 C2

C3

C4 C5 C6f

H(f)

FIR structure

x(n-1)^ x(n-2)^


x(n)^

x(n-1)^

x(n-2)^

x(n)^x(n-3)^

x(n-4)^

x(n-5)^

C0

C2

C1

C3

C4

C5

C6

C7

C8

FFT

X (m)f

poly-phase FIR structure



• More on poly-phase FX:- If many channels ~>1k required, must cascade filterbanks to

avoid excessive numbers of poly-phase FIR taps (e.g. 1 stage 1Mchannels ~10Mtaps(!); 2 stage ~2k taps).

- Each channel out of the FFT will have some leakage (aliasing) from the adjacent channel since it is critically sampled.

- Therefore, must oversample the output. Performed by “barrel-rolling” the input across phases such that the output after FFT is oversampled. The next stage poly-phase-FFT of a particular channel will therefore not contain aliasing in regions of interest so that all taken together, clean alias-free spectra are obtained…



See:

Harris, Dick, Rice, “Digital Receivers and Transmitters using Polyphase Filterbanks for Wireless Communications”, IEEE transactions on microwave theory and techniques, Vol. 51, No. 4, April 2003.

…for more detail on poly-phase filterbanks (great paper!)



• Hybrid FX-XF (a.k.a. “FXF”)- 1st stage: coarse filterbank (FIR or poly-phase FFT).

- Useful as “digital BBC” for frequency-agile sub-band placement.

- 2nd stage: XF.- Attractive as an efficient parallel processing method for

wideband signals since no large multiplier operations are required (FIRs built with memory LUTs and adders). Has larger noise-power bandwidth in sub-band so can use fewer bits connecting to the ‘X’ part.

- JVLA and ALMA correlators built this way (some slight differences in implementations). Probably the last of this breed!



• The actual signal processing operations are just one piece of the puzzle when putting a system together.

• Much of the logic and power in a system is consumed by transporting data around, synchronizing things, providing various modes of operation, error detection and recovery etc.

• Let’s look “under the hood” of the electronics…


Technology

• It all starts with fundamental physics…but moving up a level or two:- Transistor “switch”: the FET – Field Effect Transistor.


Technology

• N-MOS: applying a voltage to the Gate opens a conduction channel between the Drain and Source.

• P-MOS: applying a voltage to the Gate closes the conduction channel between the Source and the Drain.


+V (5 V)

GND (0V)

Vout

VinSimple N-MOS

inverter

MOS: Metal Oxide Semi-conductor.

MO is the insulator between the gate and the conduction channel. Extremely sensitive to electro-static discharge (ESD).

When the transistor is ON or OFF, no current flows from the gate to the conduction channel (unless it is blown…)

Current (power) only flows when changing states, to charge/discharge the gate capacitance…faster state changes consumes more power.


GND (0V)

+V (5 V)

Vout

VinCMOS inverter

CMOS: Complementary Metal Oxide Semiconductor.

Output changes faster since it is being driven both high and low.

Small amount of leakage current (power) when the conduction channel is switching states.


A 4-bit 2’s complement multiplier

F.A. (Full Adder)


Technology

• A logic gate “immediately” reflects changes on its input to its output. It can’t store a value.- A “bunch of gates doing something” is often referred to

as “combinational logic”.• A “Flip-Flop” transfers “Data in” to the output

“Data out” only on the edge of its clock:

D Q

CLOCK

Data in Data outA Flip-flop can therefore store a value…a single bit.

A Flip-flop is actually a set of cross-coupled gates with feedback implementing an “asynchronous Finite State Machine”.


Technology

• In digital electronics, pretty much everything is “synchronous”. i.e. changes occur on the clock edge all “in step”.- It’s like a production line…the speed of the line is the

clock speed and in each clock cycle each “worker” (bunch of gates doing some logic function) must get their step done before the next clock cycle starts.

- As the clock speed increases, the logic “workers” must go faster to keep up.


Technology

• Together (and in the millions/billions), Flip-flops and gates (along with memory cells) form the bulk of all digital electronics.

• As feature sizes (transistors) get smaller, more gates can be packed on a chip, they run faster, and more can be done.- JVLA correlator implemented with 90 nm and 130 nm

devices (c. 2005).- Industry currently shipping 28 nm devices…20 nm is

next…


Development process

• Fundamentally, engineering a system uses a hierarchical top-down approach:- Take a big problem, hierarchically break into smaller

and smaller problems until each small problem can be solved by a human.

- Then, put everything together, and it works! (eventually)• In reality there are several rounds of iteration and

engineering may start at any point in the hierarchy, iterate, iterate…to eventually get a coherent picture to proceed with development.


Development process

• In logic design, at the “application level”, we don’t (or, rarely) design explicitly with gates and flip-flops.

• We write HDL – Hardware Description Language code that describes logic in a high-level fashion.- And there are higher-level approaches as well.

• Can (optionally) use hierarchical graphical design tools as well to improve the human’s ability to understand how it all fits together.


Development process

• In an HDL (VHDL, Verilog), all instructions are executed concurrently (i.e. on the clock edge), and so one must think concurrently when writing code.- If some output changes “now”, it isn’t picked up until the

next clock edge.- Seems impossible to have program flow…but we create

that with pipelining and Finite State Machines.- Logic doesn’t pop in and out of existence like

functions/objects…it is always there and always active!


Development process

• “Synthesis” and “place and route” software then compiles our HDL logic design to get implemented on our target device.- For an “FPGA” – Field Programmable Gate Array, SRAM cells

in the chip are downloaded at “run time” with the compiled “program” (firmware--”bitstream”) and set the chip to execute the program by setting connections/routes/use.

- For an “ASIC” – Application Specific Integrated Circuit, the compiled program is used to plunk down gates, flip-flops, and memory and connect them with wires in the design software…which then gets built in a real chip foundry or “fab”. Wafers, die, …


JVLA WIDAR correlator

• Basic specs:- 32 antennas, 8 GHz/polarization (in 2 GHz chunks 3-bit

sampling; alternately 4 x 1 GHz 8-bit sampling).- 128 independently tunable digital sub-bands; 128 MHz,

64 MHz, downto 31.25 kHz BW per sub-band.- Each sub-band can (will be able to) have a different delay

center on the sky, within the antenna primary beam.- >60 dB (10e6:1) spurious spectral dynamic range for high

spectral purity…within XF fundamental limitations.- 16,384 to 4 million spectral channels per baseline…



• Basic specs cont’d:- “Recirculation” provides a squared increase in spectral

resolution with decrease in sub-band bandwidth.- Agile integration modes: normal, recirculation, pulsar

phase binning, burst mode.- Able to flexibly tradeoff sub-band bandwidth for spectral

resolution.- High time-resolution output; 10 msec minimum, 1 msec

possible with 10G upgrade.- Phased-array output on all sub-bands…VDIF packets, and

built-in autocorrelations.



• Basic specs cont’d:- Correlator “configuration mapper” translates high-level

observing requests from the JVLA Executor, into detailed correlator configurations.

- The config mapper gets away from “mode-based” observing, to “goal-based” observing.

- Performs mapping in flexible ways to maximize packing of resources, or maximize real-time performance (short integration times) by spreading out operations amongst available resources.



• Simplified signal processing:1. Offset the LO in each antenna by a small amount.2. Filter wideband sampled signals into integer sub-band

slots (Stage 1)…aliasing. (Other stages don’t have the integer slot restriction.)

3. Remove LO offset and “fringe phase” in complex XF correlator…causing aliasing to wash out with integration time.

4. Measure power before re-quantizer to allow wideband to be seamlessly “stitched together” from individually correlated sub-bands after correlation.



• Due to aliasing, there is a factor of 2 sensitivity loss at the sub-band edge, tapering off to none at the rate of the transition band steepness.- Using a complex FIR filter and passband overlap there

is an efficient way of eliminating this (always one more trick in the book…), but with some loss of bandwidth—the sum of all the transition band bandwidths—if implemented in the same hardware.


10e-3 passband ripple



• Sub-sample delay tracking.- Wideband delay tracking to +/-0.5 samples in memory,

produces a changing phase slope between +/-90 deg on the wideband signal.

- Within a sub-band however, the phase slope change is 1/16th of this and a changing phase that can be tracked (taken out) by the phase rotator. The residual very small amplitude and phase effects can be removed post-correlation.

- Therefore known and stable sub-sample delay correction—and with it sensitivity—is achieved, further improving stability and minimizing systematic effects.



• Due to cost and power constraints, the complex mixer/fringe rotator in the correlator chip approximates sine and cosine functions with 3 levels.- This reduces the sensitivity by ~5% and “splatters” that extra

noise across the sub-band.- Provided the chosen LO offset frequencies are not

related/don’t beat with each other, and provided the sampler doesn’t saturate to produce harmonics and intermodulation products of narrowband signals, this shows up as noise that integrates down, and generally it performs as well or better than with no fshift+mixer.


0 32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 51245

40.3

35.7

31

26.3

21.7

17

12.3

7.7

3

1.7

6.3

11

15.7

20.3

25Amplitude (dB) vs Frequency (bin)

Frequency (bin)

10logFBFq

2

0

q

No LO offset; 4-bit sampler saturating due to narrow-band tone power-to-noise power ratio (~300:1) too high (~28 dB DR).


LO offset with 3-level phase rotation; 4-bit sampler saturating due to narrow-band tone power-to-noise power ratio too high but most artifacts gone (~42 dB DR).

0 32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 51245

40.3

35.7

31

26.3

21.7

17

12.3

7.7

3

1.7

6.3

11

15.7

20.3


Frequency (bin)

10logFBFq

2

3

q


0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 25645

40.3

35.7

31

26.3

21.7

17

12.3

7.7

3

1.7

6.3

11

15.7

20.3


Frequency (bin)

10logFBFq

2

0

q

Here, 75% of power in sub-band is in the tone. No artifacts that don’t integrate down. >50 dB dynamic range…takes too long to simulate to achieve 60 dB!


S ta tio n R a c k s B a s e lin e R a c k s

Fibe

r Dem

uxS ig n a ls o n f ib e r fro m a n te n n a s : 2 7 (u p to 3 2 ) @ 9 6 G b p s e a c h

C B ES w itc h

C B EC P U

C lu s te r

M k VV L B I

R e c o rd e r

M C C C

C P C C s

B o o tS e rv e rs

-4 8 V D CP o w e rP la n t

lu s t ref ile s y s te m

-4 8 V D C -4 8 V D C

3 -p h a s e 4 8 0 V A C

H V A CU n its

(c o o lin g )

E x te rn a l n e tw o rk

c o n n e c tio n (s )

INV

1 1 0 V A C

1 1 0 V A C

P M & C

P M & C

1 1 0 V A CS m o k e

D e te c to rs

c o n tro l + s ta tu s

Mas

ter

D T S fra m e s

L T Ap k ts

V D IFp k ts

L T Ap k ts

B D F

V D IFp k ts

1 0 G E

A IR

V D IFp k ts

1 0 0 0 E1 0 0 E

1 0 0 E

1 0 0 0 E1 0 0 0 E

1000

E

1 0 0 0 E

1000

E

1 0 0 0 E

H 2 0

s ta tu s

-48

VDC

1 2 8 M H z c lo c k + e x t-T C -A (“ T im e c o d e A ” )1 2 8 M H z c lo c k + e x t-T C -B (“ T im e c o d e B ” )

M C A F1 0 0 0 E

B D F L o c In fo

10E

1 1 0 V A C

C e n tra l1 G S w itc h

M & C S w itc h e s

1 0 0 0 E

1 0 0 0 E

1 0 0 0 E

4 8 0 V A C o n U P S

H M G b p s



• Other features:- 2 banks of 2000 phase bins for high time resolution (as

low as ~12 usec for reduced spectral channels) “stroboscopic” observations of pulsars.

- Recirculation up to a factor of 256 for up to 262,144 channels on each of 16 sub-bands.

- Working on adding high time resolution burst mode for transient detection…possibly operating in the background all the time “striping” across correlator resources.


Now and the future

• Continued advances in semi-conductor technology are making correlator systems more “appliance-like” than ever before. Shipping now: 28 nm; next: 20 nm.

• A few COTS rack-mount CPUs can now do what a custom system used to have (to be engineered) to do.- The new VLBA “DiFX” correlator is a “software” correlator.

• FPGAs are more and more powerful. Latest available have ~3000 “DSP blocks”, 1-2M logic cells, useable 500 MHz clock rates, several tens of 10G and 28G transceivers…craziness!


Now and the future

• For any problem size, one has to look at capital vs operating cost (power) and decide what is the best technology. Certainly:- CPUs/GPUs for “small” to “medium” jobs. Relatively quick turnaround

time/development effort (don’t forget s/w!). Power not such a concern.- FPGAs for “medium” to “large” jobs (can easily fit the entire VLBA

correlator onto one FPGA now). Power starts to be a concern…consider ASIC migration.

- ASICs for “very large” jobs where operating cost in terms of power is of primary concern. Probably only the SKA would ever need an ASIC again.

- Poly-phase FX due to availability of large multipliers/adders and relatively inexpensive high-speed serial links.


Now and the future

• When comparing implementation concepts/ technologies and flexibility, must also bear in mind performance requirements and capabilities. - e.g. a few hundred MHz and a handful of antennas vs several

10s of GHz and hundreds or thousands of antennas.• Must also bear in mind that a fully operational, shaken-

down, facility-level system, no matter which way you cut it has software and testing overhead that takes people and time to get right.- “Wait a minute captain while I reprogram the computer to

check for sub-space frequencies…”


Questions?Thank-you

cross- correlators for radio astronomy

Documents

analog signal

weak signal

signal look like

common radio source

ratio of radio source

time domain analog

time domain digital

digital electronics