the oskar simulator (version 2!)
DESCRIPTION
The OSKAR simulator (version 2!). 3GC-II Workshop, 21 st September 2011 Benjamin Mort, Fred Dulwich, Stef Salvini http://www.oerc.ox.ac.uk/research/oskar. What is OSKAR?. Interferometer and beamforming simulator package. End-to-end s imulations of the phase 1 SKA. - PowerPoint PPT PresentationTRANSCRIPT
1
The OSKAR simulator (version 2!)3GC-II Workshop, 21st September 2011
Benjamin Mort, Fred Dulwich, Stef Salvini
http://www.oerc.ox.ac.uk/research/oskar
2
What is OSKAR?
• Interferometer and beamforming simulator package.– End-to-end simulations of the phase 1 SKA.
• Based on a full sky Measurement equation formalism.• High performance library based on NVIDIA CUDA.
3
Motivation for writing OSKAR?
• Understanding system noise limits on SKA like AA interferometers.– Dynamic range limits
• All sky simulation.• High performance required for large AA interferometer end-to-end
simulation.– e.g. ~1e6 sources, ~25 stations of ~10,000 dual polarisation antennas.
• Interferometer configuration studies with aperture arrays.
4
What I’m going to talk about!
• The OSKAR ME implementation.• OSKAR version 2.0• Some example and results.
5
Measurement Equation
• The ME as implemented by OSKAR– Scalar version currently being tested, polarised version under
development.
• K – interferometer phase.• E – Station beam.• G – Antenna element field pattern.• P – Propagation term.• B – Source brightness.• V – Complex visibility. Baseline p, q for all visible sources, s.
6
Sky Model
• The global sky model – Equatorial coordinate point source model.
• A local sky model is generated for each snapshot– Remove sources below the horizon– Transforming the source Stokes parameters to the
horizontal frame.
• Emission from bright, extended objects can be included as large collections of point sources.
7
G – Antenna field pattern matrix
• The average (fully polarized) embedded element pattern for antennas within a station
– Factored out if the antennas are sufficiently similar.– otherwise, it is absorbed into the calculation of the E-Jones term.
• Requires input from EM simulations (University of Cambridge).– Depends on the station geometry (cross coupling) and element design.
• Simple functional responses also possible.
-80 -60 -40 -20 0 20 40 60 80-50
-45
-40
-35
-30
-25
-20
-15
-10
-5
(º)
dBW
Embedded element patterns
bb
aa
gg
gg
G
8
Station Beams (E-matrix)
• Rotate phase tracking centre (=beam direction) and all sources from equatorial (RA, Dec) to horizontal (azimuth, elevation).
• Evaluate station beam response for every source and station, for the direction of interest.
• Sources far from the phase centre will be suppressed by station “primary” beams.
– (And sources far from thezenith will be suppressedfurther by antenna elementpattern.)
• Obtain complex matrix
s,iE
9
Station Phases (K-matrix)
• K-matrix effectively “phases-up” the array of stations.
• Compute phase of each source s at every station a.– Determine station (u,v,w) coordinates by rotating (x,y,z) onto a
plane perpendicular to direction of phase centre.
112exp 22,
ssi
sisi
is ikηξw
ηvξuK
10
“Correlator”
• Multiplies appropriate Jones matrices with the source brightness to obtain a complex visibility per source and per baseline, and then collapses the source dimension.
• Time-average smearing: each visibility point can be averaged over time.
– K is recomputed to include motion of baseline during integration period.– E is allowed to vary throughout the integration at a slower rate than K.
• Bandwidth smearing: multiply each visibility by fs,i,j before collapsing the source dimension.
cD
cDf
sji
sjijis /
)/sin(
,
,,,
s
s,jss,iji JBJV ,
11
The OSKAR Package
• OSKAR simulation function library based written in C making extensive use of CUDA
• Multiple libraries with simple dependencies
– Liboskar• Core CUDA function library.
– Liboskar_ms• Interface to casacore for writing simple measurement sets.
– Liboskar_apps• Utility library for using OSKAR to write C/C++ applications.
– Liboskar_widgets• Set of utility widgets written in Qt4/Qwt5. Plotting, gui components.
– Liboskar_imaging• FFT imager (CUDA based imager in development) with w-projection.
– Liboskar_fits• Interface to cfitsio for writing UVFITS and image fits files.
12
The OSKAR Package
• Simple C like interface to the main simulation library using intrinsic types where possible.
• Aims to make it possible to quickly construct new C/C++ simulation applications.
• Designed to interface easily with other languages– MATLAB either with loadlibrary() or though a MEX interface.– (Python)
• MATLAB and C/C++ applications– Beamforming simulator– AA Interferometer simulator– DFT imaging– Simple image post processing MATLAB scripts e.g. CLEAN.
13
Why CUDA?
14
Why CUDA?
• What is CUDA?
– CUDA (Compute Unified Device Architecture) is NVIDIA’s
– Program development environment based on C/C++ with some
extensions.
– Compatible with other multi threaded code.
– Multiple GPUs can be used to work together for very large problems.
• Cost and power effective desktop supercomputing.
– SIMT parallelism model.
– Requires tens of thousands of threads to be efficient.
• Multiple GPUs work together for very large problems
15
Summary of current Features / assumptions
Sky
• Equatorial system.• Point source sky model.• Support for large
numbers of sources, ~O(10^6)
• Use of point source catalogues / image pixels.
• Currently only stokes I (polarisation support very soon)
Station
• Support for any configuration of antenna elements.
• Optimised for large numbers of antennas (e.g.10,000+ per station).
• Primary beam evaluated for each station.
• Antennas currently assumed to be all identical within a station.
• EM coupling encapsulated in element pattern.
Interferometer
• Any latitude and longitude
• Any station positions.• Time-averaging
smearing by actual average.
• Analytical bandwidth smearing per source and per baseline.
• Series of visibility snapshots.
16
Some examples of using OSKAR
17
Source distribution, 2-degree “hole” at phase centre (49993 sources)
18
Field of view 2 deg across, nearest source is 1.32 degrees from centre (Telescope at 40 degrees, 480 snapshots, 49993 sources elsewhere)
Peak @ ~3E-3
19
Imperfect source subtraction
• Bright interfering source on the flank of the station beam at position X.
• A number of other sources scattered over the sky.
• Because the source has effectively become highly time-variable, a simple subtraction of its clean-component model leaves large residuals.
• limiting the dynamic range of the image.
20
Source removal
• Solving for differential gains in MeqTrees (Ian Heywood) is far more effective.
21
Simulation Example: Observation setup.
• Telescope at Faro• Pointing around
Cassiopeia.• 24h observation.
22
Simulation Example: AA station setup
• Offset grid geometry.
• ~80m diameter.
• ~2600 antennas.
23
Simulation Example: Beam pattern.
24
Simulation Example: Sky model
(Simulation with E-Jones disabled)
25
Simulation Example: Telescope setup.
26
Simulation Example: results
27
Simulation Example: Dirty image snapshots
28
Simulation Example: Frequency time source brightness profiles
(1)
(3)
(2)
29
Next Steps
• Currently working on– Polarisation.
– Efficient modelling of system noise.
– Antenna gain and phase errors.
– Limited precision numerics (currently floating point).
– Scaling up to very large simulations using multiple GPUs
• Any suggestions?