june 2012 aa power challenges workshop

WorkshopJune 2012 AA Power Challenges


Aperture Arrays

Michael Kramer


~ 250 Dishes: 0.3-3GHz

~250 Sparse Aperture Array Stations 70-450MHz

2-Core Central Region

SKA1 construction: 2016-2019

Artist renditions from Swinburne Astronomy Productions


~ 250 Dense Aperture Array Stations 400-1450MHz ~ 2700 Dishes: 0.3-10GHz



Artist renditions from Swinburne Astronomy Productions

SKA2 construction: 2020 on


Some SKA Experiments…..~ 250 Dense Aperture Array

Stations 400-1450MHz ~ 2700 Dishes: 0.3-10GHz



~ 2700 Dishes: 0.3-10GHz

DishesPrecision pulsar timing• General relativity• Gravity wavesCosmic Magnetic FieldsAstrobiology







AA‐lowEpoch of Re‐ionisation• Birth of the UniverseHI absorptionTransients






~ 250 Dense Aperture Array Stations 400-1450MHz

AA‐midBillion galaxy survey ‐ HIPulsar search and timingTransients


0

2,000

4,000

6,000

8,000

10,000

12,000

10 100 1,000 10,000 100,000

Sensitivity: Aeff/Tsys m

2 K‐1

Frequency MHz

Sensitivity Comparison

SKA2

SKA1

MeerKAT

LOFAR

ASKAP

eVLA

SKA2

SKA1

LOFAREVLA

Sensitivity Comparison

1

10

100

1,000

10,000

100,000

1,000,000

10,000,000

100,000,000

1,000,000,000

10,000,000,000

100,000,000,000

10 100 1,000 10,000 100,000Survey

Spe

ed : S

ensitivity

2 *FoV A

4 K‐2de

g2

Frequency MHz

SKA2

SKA1

MeerKAT

LOFAR

ASKAP

eVLA

SKA2

SKA1

LOFAR

EVLANote: log scale!

SKA1 & SKA2 will have much higher sensitivity & survey speed than existing instruments

ApertureArrays

Survey Speed Comparison


SKA Timeline

2019 Operations SKA1 2024: Operations SKA2

2020‐2024 Construction of Full SKA, SKA2 €1.5B

2016‐2019 10% SKA construction, SKA1 €350M

2012 Site selection

2012 ‐ 2016 Pre‐Construction: 1 yr Detailed design €90MPEP 3 yr Production Readiness

2008 ‐ 2012 System design and refinement of specification

2000 ‐ 2007 Initial concepts stage

1995 ‐ 2000 Preliminary ideas and R&D


Type of AA selection

300 500


LOFAR station


AA-low – Development


AA-low outline specification

Parameter SKA1 SKA2 Comments

Type of array Single element Single element Sparse array using a single wide‐band element

No. of elements /station 1750 11,000

No. of elements total 500,000 3,000,000 Approximately

Approx. Size of elements 1x1x2 m 1x1x2 m Must be small enough for the pitch

No. of polarisations 2 2 Each element has two receiver chains

Diameter of station 80m 180m

Number of stations 280 280 Anticipated number SKA Stations

Element communication Analogue fibre Analogue fibre Requires copper for power

Layout pseudo‐random pseudo‐random The most flexible design is as individual elements.

Frequency range 70‐450 MHz 70‐450 MHz May be down to 50MHz

Digitisation rate 1 ‐ 2GS/s 1 ‐ 2GS/s There is no frequency conversion, covers full frequency range with guard bands

Digitisation depth 6 or 8‐bit 6 or 8‐bit Required for RFI environment at these frequencies

Max instantaneous bandwidth

400 MHz 400 MHz Covers operating band of array

Output data rate /station 140Gb/s 8Tb/s Organised as 4+4bit complex data

Data rate into correlator 40Tb/s 2.2Pb/s Peta = 1015


AA-low Station

Element

AA-low StationSKA1

......

…....

Cooling

AA‐lowDigitisation &

Station Processing

RFI shielded System clock

Control &Monitoring

StationBeams

PowerGrid

Correlator&

Services

Single or dual fibres

AnalogueFibreSKA1 1,750

Elements

............

Element power distribution

Element power distribution

SKA2 11,000Elements

e/o

Powerconditioning

Elements:50-450MHz

LNA, filter, gain Power over copper

Mixer +

500MHz LO

f500MHz

Pol 2 Pol 1

Data Pol 1 & 2


SKA1 AA-low Station Processing

o/e

o/e …

o/e

o/e

RF over Fibrefrom

Elements

ADCs2GS/s6-8 bit

Spectral filters

1stBeam

forming

Station beamform

ing

o/e

o/e …

o/e

o/e

RF over Fibrefrom

Elements

Spectral filters

1stBeam

forming

1750Elements

2 x 70-450MHz

RFI shield

16Gb/sper element

140Gb/s

3 x 56Gb/sInfiniband

e/oData toCentralProcessing System

…


Uniboard 1 (Jive)– 16 input signal paths– Input bandwidth: 450 MHz, digitised– Output bandwidth: 40 beams of 300 MHz– 14 layer board; Altera Stratix 4 230: 500 GMAC/s


Uniboard 1 Implementation

ADC Interface

Shelf:• 4 Processors• 8 ADC interfaces• 64 inputs

(32 elements)


Analogue and Comms PowerProcessing requirement

Processing and digitisation

AA-low SKA1 Station power

Spectral filter:Polyphase filter into 1024 channels 105 MACsPFF rate at 1GS/s 106 /sProcessing rate per element 2*1011 MAC/sTotal spectral filter proc. (1750 el.) 3.5*1014 = 350TMAC/s

Beamforming:Each element 40GS/s (>160Gb/s): 8*1010 MAC/sTotal processing/station (1750 el.): 1.4*1014 = 140TMACs

Total station processing: ~500TMAC/s

Technology FPGA(TMAC/s)

Board (TMAC/s)

# per station*

Power/Board, inc ADC (W) Total

UNIBOARD 1 0.5 4 200 400 80kW

UNIBOARD 2 ~4.0 32 25 500 12.5kW

SKA1 processing 10 est. 80 10 700 7kW*allowance made for inefficiency

Element powerLNA 50mW 100mWGain chain and mux 50mW 100mWOptical Transmission 100mw 150mWTotal Element power 350mWAll elements <1000W

Communications etc. powerTransmission 3*56Gb/s 100WInternal comms 30*56Gb/s 300WMisc. 1000W

Total Station 2.5kW

Total AA-low station power ~10kW 3MW tot


Functions:Correlator, beamformer, gridding, imaging, searching, timing & storage

Flexible, upgradeable, development path

Central processing pipelines

…IncomingData fromcollectors

Switch

Buffer store

Switch

Buffer store

HPC BulkStore

CorrelatorBeam

former

UV

Processor

Imaging:

Non‐Imaging:

CornerTurning

CourseDelays

Fine F‐step/Correlation

VisibilitySteering

ObservationBuffer

GriddingVisibilities Imaging

ImageStorage

CornerTurning

CourseDelays

Beamforming/De‐dispersion

BeamSteering

ObservationBuffer

Time‐seriesSearching

Searchanalysis

Object/timingStorage


Processing rack and blade

Host processorMulti‐core X86

GPU

‐1 ‐>10TFLOP/s

GPU

‐2‐>10TFLO

P/s

To rackswitches

Disk 1≥1TB

56Gb/s

PCI Bus

Disk 2≥1TB

Disk 3≥1TB

Disk 4≥1TB

Processing blade 1Processing blade 2Processing blade 3Processing blade 4Processing blade 5Processing blade 6Processing blade 7Processing blade 8Processing blade 9Processing blade 10

Processing blade 11Processing blade 12Processing blade 13Processing blade 14Processing blade 15Processing blade 16Processing blade 17Processing blade 18Processing blade 19Processing blade 20

Leaf Switch‐1 56Gb/sLeaf Switch‐2 56Gb/s

42U Rack

Processing Blade:

MIC?

• 20 TFlop• 2x56 Gb/s comms• 4 TB storage• <1kW power

• Capable host (dual Xeon)

• Programmable• Significant RAM

Blade Specification

21


Central Processing System

1280 Blades incl. correlator

…

…

…

…

…

…

…

…

…

…

…

…

AA-lowData 1

1

280

AA-lowData 2

1

280

DishesData 4

1 2 16

… 1 3 N

…

HPC

BulkStore

2

Switch

Correlator/UV processor

Further UV processors

Imaging Processor

Corner Turnerswitches

56Gb/s each

…

…

AA-lowData 3

1

280

1

250

……

16 switches36 x 56Gb/s ports


Central processing power

Racks Power kW Comments/rack Total

Corner turning switches 2 7 14 Houses both AA and Dish input switches

Correlator/UV processor 48 31 1488 Correlator with UV processing capacity

UV processor per 64GB/s 16 31 496 Number of UV processors to support 64GB/s of visibilities

Central switch 1 10 10 Occupies 29U, but expect rest will be filled with something

Imaging processor per PFLOP/s 5 80 400 Assumes BG/Q, more processing scales linearly

Bulk store with 4.3PB 3 20 60 1.5PB/rack expansion

Total for Basic system 75 2,468 kW

Includes AA-low & dishes


SKA Phase 2……

Challenges:• Very high data rates• Large numbers of elements• Substantial processing requirements

100MW !


SKA2 wide area data flow

16 Tb/s 4 Pb/s

24 Tb/s

20 Gb/s

20 Gb/s


Elements Bunker

AA-low signal path

LNA

Antenna

GainBlock

Analogconditioning

AnalogueSignal Transport

ADC

ADC – Processor comms

Digitisation P

rocessing

Tile – station processoroptical comms

Prim

ary Station P

rocessing

Secondary Station Processing

Wide areaoptical comms

To Correlator

Digitisation Station Processing

optical interconnect

Clock Distribution

e/o o/e


SKA2 AA Station processor

PrimaryStation

ProcessorBoard

0

…..

PrimaryStation

ProcessorBoard

1

…..

PrimaryStation

ProcessorBoard

(max 35)

…..

…..

SecondaryStation

ProcessorBoard

0

SecondaryStation

ProcessorBoard

1

SecondaryStation

ProcessorBoard

(max 35)

…..

…..

…..

…..

…..

…..

…..

012

35

ToElement

Digitisation

Long distance drivers

…..

012


…..


…..

Optical linksTo Correlator

Each link is 12 fibre lanes@10Gb/s

“All to All”Connections


…..

…..

…..

Station Processor

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

012345

303132333435


To Element digitisationor Primary

Station Processors

To Secondary Station Processors or long distance fibre drivers


Each link is 12 diff. copper lanes@10Gb/s

12-channel Rx module. e.g Avago

AFBR-820BXXZ

12-channel Tx module. e.g Avago AFBR-810BXXZ

Total Raw input data rate: 4.32Tb/s

Total Raw output data rate: 4.32Tb/s max

“All to All”Connections

ControlProcessor

LineTx/R

x

Station Control

Requirements:• High bandwidth in • High bandwidth out• Largely cross connected• Scaleable at various levels• Programmable beamforming

PChip20

TMAC

6 x

120G

b/s

6 x

120G

b/s25 watt


SKA2AA-lowPower

Subsystem SKA2 RemarksAntenna element: mW

1 LNA 50 Projection from SKADS work2 Antenna gain block 40

Digitisation and processor:3 Analogue sig conditioning 40 Estimate4 ADC 75 1.0 GS/s each channel5 Clock distribution 10 estimate (less with more ADC per chip)6 Coms: ADC to processor Phase 2: integrated ADC and processor7 Digitiser Processor 300 SKA2: 40 W for 128 receiver inputs8 Digitiser Control ccts etc. 140 SKA2: 25 W for 128 receiver inputs9 Copper comms: Processor to optical driver 10 1.2 W for 128 receiver inputs

10 Optical Coms: Tile to Station proc. 60 SKA1: 4.4 W. SKA2: 2.5 W for 40 receiver inputsStation Processor:

11 Primary Station processor 75 SKA2: 600 W. for 7776 receiver inputs12 Copper comms:

Processor to optical driver10 1.2 W for 128 receiver inputs

13 Optical Coms:Primary to Secondary Station proc.

20 SKA2: 2.5 W for 40 receiver inputs

14 Secondary Station processor 75 SKA2: 600 W. for 7776 receiver inputs15 Copper comms: Processor to optical driver 10 16 Tb/s station output with

153,000 receivers @ 10mW/Gb/sLong Distance comms:

16 Wide area comms Accounted for separately

Electrical power used 915Electrical power supplied @ 85% efficiency 1,076Power incl cooling at 25% cooling power 1,345

Total station power 31 kW


AA-mid Array – SKA2


Possible AA-mid construction

Beamformer

......

......

......

......

Beamformer

The join!

Membrane

Ground plane

Guideframe

Tile supportGround

Top View

......

......Non-conductingGuideframe


Vivaldi

AA-mid elements

FLOTT: (a)(d)

BECA: (b)(e)

ORA: (c)(f)


AA-mid proposed signal path

LNA

Antenna

GainBlock

AnalogCond.

Local Tile Analogue Signal Transport

ADC

ADC – Processor comms

Tile Digital Processing

Tile – station processoroptical comms

Primary Station Processing

Secondary Station Processing

Wide areaoptical comms

To Correlator

Tile Processing Station ProcessingFront-end

optical interconnect

Clock Distribution

RF B

eamform

ing

Tile

Tile

Tile

TTD 4-element clusters

Tile processingunit

Element & gain. Phantom power


AA-mid designParameter Value Comments

Type of array Single element Dense array using Vivaldi or ORA.

Number of elements 110,000

Pitch of elements 15 cm λ/2 at 1000MHz

No. of polarisations 2 Each element has two receiver chains

Diameter of station 56m

Cluster size 4 elements Uses true time delay beamforming

Tile size 16 x 16 elements Built out of 4 x 4 clusters

No. of Tiles 430 Each tile is ~2.4m square

Number of stations SKA2 250 Anticipated number of Phase 2 SKA Stations

Element communication Copper Includes Phantom power

Layout Dense rectangular Regularly spaced

Frequency range 400‐1450 MHz

Digitisation rate 3GSamples/s There is no frequency conversion,

Digitisation depth 6/8‐bit Required for RFI environment at these frequencies

Beamforming technology Digital Using cluster outputs

Max inst. bandwidth 1000 MHz Covers operating band of array

Max output data rate 16Tb/s Organised as 4+4bit complex data


AA-midPower

Subsystem SKA2 RemarksFront‐end (Quads of 4 elements): mW

1 LNA total 120 4xLNAs at 30mW each2 Antenna gain block 120 4 units at 30mW each

Tile beamformer:3 Analogue sig conditioning 160 4 channels at 40mW each4 Analogue beamforming 100 Combiner for 4 elements for digitisation5 ADC 50 2.5 GS/s, 10Gb/s each channel6 Clock distribution 10 estimate (less with more ADC per chip)7 Coms: ADC to processor Integrated ADC and processor8 Tile Processor 200 25 W for 128 digital channels9 Tile Control ccts etc. 100 13 W for 128 digital channels

10 Copper comms: Processor to optical driver 10 1.2 W for 128 digital channels11 Optical Coms: Tile to Station proc. 20 2.5 W for 128 digital channels

Station Processor:12 Primary Station processor 130 600 W for 4608 digital channels13 Copper comms:

Processor to optical driver10 1.2 W for 128 digital channels

14 Optical Coms:Primary to Secondary Station proc.

20 2.5 W for 128 digital channels

15 Secondary Station processor 130 600 W for 4608 digital channels16 Copper comms: Processor to optical driver 10 16 Tb/s station output with

153,000 receivers @ 10mW/Gb/sLong Distance comms:

17 Wide area comms Accounted for separately

Electrical power used 1,190 Per digital channel Electrical power supplied @ 85% efficiency 1,400Power incl cooling at 25% cooling power 1,750

Total station power 96 kW


2018+ Technology Shopping listTsys for AA-mid <40K

Analogue system power 100mW per Rx

>3GS/s ADC power <100mW

DSP processor chip performance >20TMACs

DSP Power for 20TMAC ~25W

DSP chip comms, I/O count >128 x 11Gb/s

DSP-digitiser integration Possible

50m optical links, pluggable >120Gb/s

50m 120Gb/s link power 2.5W

Flash storage module capacity 20TB

Many-core streaming processor >50TFlop

50TFlop processor power <300W

Super computer performance >10PFlop

Super computer power <1MW/PFlop

AA Power challenges

can be met with realistic

developments*

* e.g. International Technology Roadmap for semiconductors, ITRS

june 2012 aa power challenges workshop

Documents