IceCube simulation with PPC

Photonics: 2000 – up to now Photon propagation code PPC: 2009 - now

Photonics: conventional on CPU

• First, run photonics to fill space with photons, tabulate the result

• Create such tables for nominal light sources: cascade and uniform half-muon

• Simulate photon propagation by looking up photon density in tabulated distributions

Table generation is slow Simulation suffers from a wide range of binning artifacts Simulation is also slow! (most time is spent loading the tables)

PPC simulation on GPUgraphics processing unit

execution threads

propagation steps(between scatterings)

photon absorbed

new photon created(taken from the pool)

threads completetheir execution(no more photons)

Running on an NVidia GTX 295 CUDA-capable card,ppc is configured with:

384 threads in 33 blocks (total of 12672 threads)average of ~ 1024 photons per thread (total of 1.3 . 107 photons per call)

Direct photon tracking with PPC

• simulating flasher/standard candle photons• same code for muon/cascade simulation

• using Henyey-Greenstein scattering function with <cos >=0.8• using tabulated (in 10 m depth slices) layered ice structure• employing 6-parameter ice model to extrapolate in wavelength

• transparent folding of acceptance and efficiencies• precise tracking through layers of ice, no interpolation needed

• much faster than photonics for E-2 nugen and unweighted CORSIKA: 1000-1200 corsika files (4 sec each) in 24 hours 2500-3500 E-2 nugen files in 24 hours IC-40i.e., 10000 E-2 nugen files in ~3-4 days on 6-GPU cudatest

in ~1 day on 3 cuda00X computers

photon propagation code

Photon Propagation Code: PPCThere are 5 versions of the ppc:

• original c++• "fast" c++• in Assembly• for CUDA GPU• icetray module

All versions verified to produce identical results

comparison with i3mcmlhttp://icecube.wisc.edu/~dima/work/WISC/ppc/

ppc-gpu

ppc icetray module

• moved from sandbox to http://code.icecube.wisc.edu/svn/projects/ppc/trunk/

• outdated code removed (old slower code that has not changed since 09/09)

• new code has a wrapper: private/ppc/i3ppc.cxx, which compiles by cmake system into the libppc.so

• it is necessary to compile an additional library libxppc.so by running make in private/ppc/gpu:

“make glib” compiles gpu-accelerated version (needs cuda tools) “make clib” compiles cpu version (from the same sources!)

• link to libxppc.so and libcudart.so (if gpu version) from build/lib directory

• this library file must be loaded before the libppc.so wrapper library

Should perhaps compile the CPU version by default by cmake system?

Configuration filesppc needs the following tables:

wv.dat wavelength-tabulated DOM acceptance calculated from qe_dom2007a table of efficiency.h fileas.dat overall DOM efficiency and parameters of the angular sensitivity polynomial expansion

before running the program create a link to one of the provided files: as.nominal nominal (measured in the lab) as.holeice hole ice (corrected by hole ice model)

rnd.txt table of random number multipliers for the multiply-with-carry random number generator

tilt.par files describing the ice tilttilt.dat delete unless using icemodel.sp2 (SPICE^2)

icemodel.par file with 6 parameters of the icemodelicemodel.dat main ice properties table: depth/b_e(400)/a_dust(400)/\delta\tau (as in report icecube/200911002)

before running the program create a link to one of the provided files: *All models* to be used with as.holeice unless otherwise specified icemodel.aha AHA model icemodel.sp1 SPICE model icemodel.sp2 SPICE^2 model icemodel.sp2+ SPICE^2+ model icemodel.sp2+n SPICE^2+ model with nominal ice icemodel.sp2x SPICE^2x model **ATTENTION** You must delete files tilt.par and tilt.dat if using AHA or SPICE, but not SPICE^2 Only SPICE^2 and higher have been fitted to use the tilted ice description.

Can specify alternative directory containing these tables with the PPCTABLESDIR env. variable: this could be a part of simprod or a separate project (for consistency), or part of ppc

Other configuration parameters

A few configuration parameters may be set as “#define”s in private/ppc/gpu/ini.cxx file:

#define OFLA // omit the flasher DOM#define ROMB // use rhomb cells aligned with the array

#define ASENS // enable angular sensitivity#define TILT // enable tilted ice layers#define MKOW // use Marek Kowalski's photon yield parametrization

#define MLTP 30 // number of multiprocessors#define WNUM 33 // optimized for 30 multiprocessors#define OVER 10 // size of photon bunches along the muon track#define NTHR 384 // NTHR*NBLK should not exceed the count of different random number multipliers

#define NPHO 1024 // maximum number of photons propagated by one thread#define HNUM 1048576 // size of the output hit buffer, must hold hits from up to NPHO*NTHR*NBLK photons

#define MAXLYS 180 // maximum number of ice layers#define MAXGEO 5200 // maximum number of OMs

#define OVR 5 // over-R: DOM radius "oversize" scaling factor

Outlook

• ppc has been used to simulate lots of IC40 data with V02-03-00 of simulation simulates 1 10000-file nugen E-2 set (sufficient for an entire analysis) in 1 day on 3 cuda001/2/3 computers (18 GPUs)

• need to: verify that it works for V02-04-00 of simulation add code to treat high-efficient DOMs correctly verify that it works for IC59 improve flasher simulation (interface with photoflash) figure out the best way to compile