New Zealand Involvement in Solving the SKA Computing Challenges

D R A N D R E W E N S O R

D I R EC TOR H P C R ES EARC H L A B O R ATORY/ D I R EC TOR N Z S K A A L L I A N C E

CO M P U T I N G FO R S K A CO L LO Q U I UM 2 0 1 8

1 5 F E B R UA RY 2 0 1 8

NZ Background in the SKANZ was a founding member of the SKA Organisation

NZ had joint bid with Australia to host SKA until dual site decision in 2012

First substantial involvement by NZ in a mega-Science project

SKA Board full member representation by MBIE

MBIE desire to focus on computing/software aspects of project

NZ SKA Alliance (NZA) formed in 2013 to coordinate NZ’s preconstruction contributions to SKA

Joined the Central Signal Processor (CSP) and Science Data Processor (SDP) consortia in 2013

Will be longest term multiway academic-industry collaboration

Possibly largest NZ involvement in international ICT collaboration

NZ SKA Alliance OrganisationsNZA has seven member organisations, six with contracted design work

Currently active in Preconstruction Design Active in part Associated organisations

NZA People

• Over 35 academics and industry partners in NZ working on SKA design for Central Signal Processor and Science Data Processor

• Numerous NZA people fulltime on SKA• Team still grew during 2017 to handle preconstruction workload• Includes all of NZ’s involvement in SKA1 preconstruction• One of largest SKA computing groups worldwide, very strong international collaborations

SKA1 Data Flow and International Consortia TeamsMID

LOW

SKA1 Data RatesBiggest BIG data project in the world

◦ SKA1 has BIG Data on a scale that ICT Industry will befacing in 5-10 years time

◦ SKA2 will be an order of magnitude BIGGER yet

Raw data rates◦ Dishes produce around 2 TB/s

◦ Antennas produce around 157 TB/s (0.3 TB/s per station)

Central Signal Processor◦ Real-time processing of data on the fly

◦ All data processed together in correlator, outputs up to 0.7 TB/s

◦ Pulsar Search pipeline must sift through data every few minutes to find new candidate pulsars

Science Data Processor ◦ Processes correlator output to form images of the sky will require approx 260 PetaFLOP machine

◦ Mix of real-time and off-line processing, a lot of iterative processing on data

Central Signal Processor

Channeliser

Correlator

Beamformer

Pulsar Search

Pulsar TimingLocal

Monitoring and Control

Signal data from Mid dishes/Low stations

Visibility data to SDP Imaging/ Calibration

Telescope Manager

Pulsar Search candidates to SDP

Pulsar Timing data to SDP

CSP Correlators and BeamformersMid CBF

◦ Led by NRC Canada◦ Contributions from NZA (AUT, Massey) and INAF◦ “Frequency slice” two stages of custom FPGA boards, large data movement challenges◦ NZ involvement in FPGA firmware, ASIC backup design◦ See talks by Will Kamp and Rezaul Hasan◦ CDR documentation submitted, awaiting review in March

Low CBF◦ Led by CSIRO◦ Contributions from NZA (AUT) and ASTRON◦ Custom FPGA boards, large data movement challenges◦ NZ involvement in project management, modelling, FPGA firmware◦ See talks by Grant Hampson and David Wilson/Norbert Abel◦ CDR documentation submitted, awaiting review in March

CSP Pulsar Search◦ Pulsar Search led by Manchester

◦ Contributions from Oxford, Max Planck, NZA (Auckland), INAF, STFC

◦ Same design for Mid and Low telescopes

◦ Commodity GPU and FPGA boards, large compute challenge and a lot of hardware

◦ NZ involvement in FPGA OpenCL software

◦ See talk by Oliver Sinnen

◦ CDR subelement review passed in Jan 2018

CSP Pulsar Timing◦ Pulsar Timing collaboration between Swinburne and AUT

◦ Same design for Mid and Low telescopes

◦ Commodity GPU, a lot of software but small hardware

◦ NZ involvement in project management, algorithm and software design

◦ See talk by Willem van Straten

◦ CDR subelement review passed in Jan 2018

Central Signal Processor Organisations

NZA is third largest worldwide in CSP

0

100

200

300

400

500

600

700

800

900

ASTRON CSIRO IANF MPIfR NRC NZA OUX STFC SWINB UMAN

Reported Hours per Month (taken from 2016 but still approximately true)

Science Data ProcessorCalibration

gains/ residuals

Local Monitoring and Control

Telescope Manager

Visibility data from CSP Correlator Gridding (Inverse) FFT Cleaning

FFTDegridding

Pulsar Search candidates from CSP

Pulsar Timing data from CSP

Candidate Sifting

Timing Analysis

Sky image/ updated model

Sky model

SDP Imaging Pipeline◦ Contributors include Manchester, NZA (AUT), Cambridge, ASTRON, CSIRO, SKA South Africa

◦ Same design for Mid and Low telescopes

◦ Extremely large compute and data movement challenges

◦ Together with calibration forms bulk of SDP processing

◦ NZ involvement in algorithm optimisations on accelerators, modelling, precision analysis

◦ See talks by Seth Hall and Anthony Griffin

SDP Compute Platform◦ Contributors include Cambridge, Oxford, ASTRON, NZA (AUT, Catalyst, Compucon, Open Parallel), CSIRO,

ICRAR, China

◦ Same design for Mid and Low telescopes

◦ Extremely large amount of hardware, system orchestration, data movement, compute, data buffering

◦ NZ involvement in node design, prototyping/benchmarking, middleware, services, OS, security

◦ See talks by JC Guzman, Drew Devereaux, Markus Dolensky, TN Chan, Piers Harding, Nicolas Erdody

Science Data Processor Organisations

NZA is third largest worldwide in SDP

Reported hours per month totalled for 2017

Questions?

Thank You

Andrew Ensor

Director HPC Research Laboratory/ Director NZ Alliance

andrew.ensor@aut.ac.nz

+64-9-9219999