Study of the core-cusp problem in cold dark matter halos using N-body simulations on GPU clusters

()Hello, everyone. I am Go Ogiya from U-Tsukuba, Japan.I am very happy to be here and to talk about our study.Id like to thank all of you for giving me such a wonderful opportunity.The topics of my talk are speed-up of the tree method by using GPU cluster and the dynamical response of DM halos to oscillatory change of gravitational potential. (The title of my talk is Study of the core-cusp problem in cold dark matter halos using N-body simulations on GPU clusters, and they are my collaborators.)These are my collaborators. Masao Mori was Andis post doc. fellow 13 or 14 years ago, and he is my supervisor.1ContentsIntroductionThe Core-Cusp problem (GO & Mori; arXiv1206.5412)Change of galactic potentialAnalytical modelResults of simulationsSummary & Discussion

This is the topics of todays talk.At first, I will introduce the scientific background of our research.The main topic of this talk is the dynamical response of DM halos to oscillatory change of galactic potential.I will explain an analytical model to discuss the dynamical response of DM halos, and show you the results of N-body simulations.

After this topic, I will talk about our code, if we have time. We are developing a tree code for GPU clusters.

2Ishiyama et al. (2012)

What is the Core-Cusp Problem? Navarro et al.(1997) Mass-density profile of DM haloObservation vs Theory (CDM)

log() [10-3 M pc-3]-1 0 1 log(r) [kpc]van Eymeren et al. (2009)See also Burkert (1995)Constant : Core

log(r) [kpc]

log() [ 1010M kpc-3]Divergent : Cusp For structure formation in the universe, dark matter plays crucial role.It is the main driver for structure formation.The present standard paradigm is the Cold Dark Matter cosmology.Its prediction is well matched to observational results in large scale, but there are sore serious problems in small scale.On of them is the Core-Cusp problem.Many observations have reported that DM halos with less massive galaxies have constant mass-density around the center. This is called Core.Andi has found the existence of the core structure early, and has pointed out the relation between core scale and halos mass.On the other hand, cosmological N-body simulations based on CDM cosmology have predicted that mass-density of CDM halos are divergent, and this structure is called Cusp.This discrepancy between observation and theory is well known as the Core-Cusp problem.Effects of Mass-LossThe central cusp becomes flatter when mass-loss occur in short timescale.But the central cusp still remains.

Initial conditionGO & Mori (2011)Before todays main topic, I will show you the results of our previous work.In this paper, we have studied the dynamical response of DM halos to galactic winds driven by supernova feedback.This corresponds to the cases in which supernova feedback is powerful, and galactic gas has been ejected from galaxies.To investigate it, we have performed N-body simulations parameterizing the timescale of gas mass-loss. This is related to the SFHs of galaxies.This figure demonstrates the semi-equilibrium states after gas mass-loss. And it shows the central cusp becomes flatter when mass-loss occur in short timescale, however, the central cusp still remains.Thus we have concluded the mass-loss is not an efficient mechanism to flatten the central cusp of DM halos.

Realistic SimulationMashchenko et al. (2008)Cosmoligical N-body+SPH simulationSupernova feedback etc.Blue: gas, Yellow: starGas Blown out (expansion)Fall back towards center (contraction)Repeat many timesGas OscillationCusp-Core transitionPhysical mechanism???

Motivation: understand the mechanism for cusp flattening Here, I will show you a movie from Mashchenko et al. 2008.They have performed a cosmological hydrodynamical simulation.Supernova feedback is taken account in this simulation. Blue and yellow points represent gas and star, respectively.Please take notice gas component, blue points.Gas is blown out from galactic center by supernova feedback.Subsequently, it lose energy and fall back towards the center of galaxy.And supernovae expel gas component from the center again.These phenomena are called galactic fountains or galactic breathing.They reported that the central cusp has flattened out.But the physical mechanism is still unclear.Idealized Model1) Gas heating by supernovae3) Energy loss by radiative cooling4) Contraction towards the center5) Ignition of star formation againRepetition of these processesGas OscillationChange of potential DM halo is affected gravitationally Cusp to Core transition?2) Gas expansionDM haloGasThis is a schematic picture expressing a scenario to flatten the central cusp.This is the idealized model of the result of previous movie.Firstly, supernovae occur in the galactic center and heat gas.Secondly, heated gas expands.And it lose energy by radiative cooling, contracts towards the center subsequently.Finally, star formation activity will ignite again.We consider repetition of these processes or gas oscillation.Gas oscillation makes change of galactic potential, and affect DM halo gravitationally.We suppose Cusp to Core transition may occur during gas oscillation.

Linear analysis: Resonance model

To discuss the dynamical response of DM halos to oscillatory external force analytically, we have done a linear analysis of the resonance between the density wave and the particles.We consider the situation in which the equilibrium system, labeled 0 is perturbed by the external force, and some physical quantities are induced.We focus on a particular group of particles has rho_0, and v_0.And we assume the external force has a oscillatory form like this.We have solved linearized Eulers eq. and eq. of continuity, and have derided the solutions.These are just like the solutions of forced oscillation problem of harmonic oscillators.We notice that coefficients diverge when Omega, frequency of external force is similar to kv_0.This indicates the occurrence of Landau resonance.Resonance between particles and density waves

Some arithmetic calculations (n = 1)

CDM mass-density:

Resonance condition for CDM haloResonance occurs when the condition is satisfied efficient energy transfer system expands Cusp-Core transition

Gausss hyper-geometric functionSo our answer for the question What is the physical reason of our simulation results ? is resonance between DM particles and gas oscillation.The resonant condition can be rewritten by this form.The density will change dramatically when the condition is satisfied.So the resonance will flatten out the central cusp of DM halos. Furthermore, our model can be connected some observed values.The oscillation period in our model corresponds to star formation histories of galaxies, and local dynamical time relates to density of DM halos.

Numerical ModelBaryon (external potential):Hernquist potential (Hernquist 1990)DM halo (N-body system):NFW model(Navarro et al. 1997)

Property of DM halo

Oscillation period of the external force, T

To test this hypothesis, we have performed N-body simulations.N-body system represents a DM halo and is an equilibrium Navarro-Frenk-White model, initially.To express baryon component, we adopt external Hernquist potential, and we change its scale length in a oscillating manner to represent gas oscillation.Fundamental parameters are listed in this table.We focus on the oscillation period, T as an important parameter.N-body simulations have been performed on GPU cluster, HA-PACS in CCS of U-Tsukuba.

Results1 -Density Profile-The cusp-core transition and the resultant core scale depends on the oscillation period of the external potential, T.Initial condition

after 10 cycles This is the resultant density profile of DM halos after 10 oscillation cycles.The horizontal and vertical axes mean the distance from the center, r, and mass-density of DM halo. Each line represents the result of different oscillation period, T.Yellow one is result of high resolution run of T=3td. The solution is well converged.It is clear that the cusp-core transition and the resultant core scale depends on the oscillation period of the external potential, T.And vertical dashed lines mean the core scale predicted by our analytical model.Our model have predicted the resultant core scale precisely.

Initial condition

Fourier spectrum of radial velocity

Peaks appear when =2/T .

Each position of the peaks matches the core scale.Result2 -Fourier Spectrum of Velocity-Density profiles of DM halosafter several oscillation periods

r [kpc]

Resonance&Core creationWe also analyze the Fourier spectrum of radial velocity of DM halos.It is demonstrated in the lower panel.In this panel, the Fourier spectrum of radial velocity of the system is shown as a function of r. Peaks appear when we choose omega=2pi/T.And each position of the peaks matches the core scale.From these results, DM halos are accelerated through resonance and core structure has been created.Result3 -Overtones-=n Spectrum with peak (Resonance)

r [kpc]351052111Our analytical model also predicts the resonance of overtone modes. To verify it, we have analyzed the Fourier spectrum of radial velocity for various frequency.Each line represents the result of same run as previous figures.At the top of each panel, extracted frequency is shown and each appended number represents the index of overtones, n.So, for example, in upper left figure, panel of omega = 2pi / 1td, the resonance of fundamental-tone for T=1td run appears.And the resonance of 3rd overtone for T=3td run and 10th overtone for T=10td run also have arisen.In the same manner, in this panel (omega=2pi/2td), 5th overtone for T=10td run appears.As expected, the resonance of overtone modes have occurred in our simulations.Summary -Core-Cusp problem-Study the dynamical response of a DM haloThe Core-Cusp problem Oscillatory change of gravitational potential

Analytical ModelResonance between particles and density waves Resonant condition: dynamical time oscillation period

N-body simulationsResonance plays a significant role to flatten central cuspResonance Efficient energy transfer Cusp to Core transitionCore scale is well matched to our predictions

I will summarize this talk.To resolve the Core-Cusp problem, we have studied the dynamical response of DM halos to oscillatory change of gravitational potential.We have constructed an analytical model to discuss the resonance between DM particles and density waves and have found the resonant condition. Furthermore, we have performed collisionless N-body simulations to test our prediction and have obtained the results resonance between DM particles and gas oscillation around the center plays a crucial role to flatten the central cusp.Thank you for your attention!Discussion -Link to Observations-If DM halos resonate with their host galaxies, resonant condition may be satisfied in observed galaxies.

SFHs of galaxies and density profiles of DM halosex. Holmberg II

(Oh et al. 2011)McQuinn et al. (2010)Regarding as an oscillatory SFH, T ~ 100Myr

If DM halos resonate with their host galaxies, resonant condition may be satisfied in observed galaxies.Oscillation period, T can be related to the SFH of observed galaxies, and the local dynamical time, td corresponds to the density of DM halos.Our model implies that these two observed values may correlate with each other.As an example, I exemplify a dwarf irregular galaxy, Holmberg II.Its SFH is derived by McQuinn+, and regarding it as an oscillatory SFH, the period is about 100Myr.The density profile is shown in Oh+, and they indicate that Holmberg II has the core structure around the center.We calculate the dynamical time at the core scale, about 0.5 kpc, and obtain td is about 50 Myr.It is quite similar value to the period of SFH.The resonant phenomena might have occurred in this galaxy.We hope to study about this correlation for many sample galaxies as a subsequent work. Structure Formation in the UniverseDark Matter (DM)is the dominant element in mass interacts only through gravity with othersassembles baryon (atoms) and DMdrives structure formation (DM halos, Galaxies etc.)

Cold Dark Matter (CDM) cosmologyis the standard paradigm matches observations in large scale has serious problems in small scale

Core-Cusp problem

Ishiyama et al. (2011)For structure formation in the universe, dark matter plays a significant role.Upper The present standard paradigm is CDM cosmology.Its predictions match observational results in large scale, but it has some serious problems in small scale.One of them is our research target, the Core-Cusp problem.Resonance Model

Resonance

To discuss the dynamical response of DM halos to oscillatory external force analytically, we have done a linear analysis of the resonance between the density wave and the particles.We consider the situation in which the equilibrium system, labeled 0 is perturbed by the external force, and some physical quantities are induced.We focus on a particular group of particles has rho_0, and v_0.And we assume the external force has a oscillatory form like this.We have solved linearized Eulers eq. and eq. of continuity, and have derided the solutions.These are just like the solutions of forced oscillation problem of harmonic oscillators.We notice that coefficients diverge when Omega, frequency of external force is similar to kv_0.This indicates the occurrence of Landau resonance.Resonance between particles and density waves

Some arithmetic calculations (n = 1)

Resonance ConditionResonance occurs when the condition is satisfied efficient energy transfer system expands density change dramatically Cusp to Core transition

So our answer for the question What is the physical reason of our simulation results ? is resonance between DM particles and gas oscillation.The resonant condition can be rewritten by this form.The density will change dramatically when the condition is satisfied.So the resonance will flatten out the central cusp of DM halos. Furthermore, our model can be connected some observed values.The oscillation period in our model corresponds to star formation histories of galaxies, and local dynamical time relates to density of DM halos.

Core ScaleDensity profile of CDM halo

Mass profile

Resonant condition Dynamical time

Gausss hyper-geometric function

Here, I will apply our analytical model to the halo mass-profile models predicted by cosmological N-body simulations.The density structure of DM halos is well fitted by this formulation and mass profile is given like this.Using the resonant condition we derived and the definition of local dynamical time, we obtain the formulation to predict the core scale created by resonance.Giving some parameters, virial mass, oscillation period and so on, we can predict the resultant core scale.Resonant SolutionWhen the condition is satisfiedUsing l'Hpital's rule

Andi gave me two homework.Homework 1 is about the relation between oscillation period, T and resultant core scale, rcore.From our analytical model, we have derived this expression to predict the core scale as a function of T. In this figure, we compare the prediction with simulation results.In the cases of rapid oscillation, our model have predicted the core scale perfectly.However, in the run of T=10td, prediction is less than successful.Actually, in the derivation process of this expression, we assumed rcore is sufficiently smaller than scale length of DM halo, Rdm.Thus this is not suitable for the run of T=10td, unfortunately.

Relation between T and

NFWThe second homework from Andi is about the relation between oscillation period, T and the power law index of resultant density profile.For the resultant density profile of DM halos after 10 cycles after, I derived the power law index of the center.However, they will change depending on the cycles of oscillation as I have demonstrated.The core scale will be saturated since the number of cycles to exceed potential energy diverge.

Discussion Energy Transfer Rate-

In this slide, we evaluate the energy transfer rate from the external force to the system.Firstly, using the expressions for the external force and density, we average the force on the system over a wavelength, and obtain the spatially averaged force, like this.Here, we have neglected the interactions between different Fourier modes.

Secondly, we evaluate the energy transfer rate from the external force to the system by integration for the velocity space, and derived this equation.Where f means the distribution function of velocity.

W/(dK/dt)r [kpc]We can predict the core scale by using this argument.Here, we compute the energy transfer rate by adopting distribution function of the equilibrium NFW model. Upper figure shows the number of cycles to exceed potential energy as a function of r.In our simulations, the number of cycles is set 10.According to our model, inputted energy by the external force will be exceed initial potential energy in these region.We regard that the core/bump structure will be created there, below this line.For the run of T=1 and 3td, we have predicted the resultant core scale quite precisely.Our model tends to predict core scale slightly larger than simulation results.It is due to non-linear or multi-dimensional effects. They are not taken account into the model.For the run of T=5 and 10td, the model failed the prediction.It predicts the core creation, but the central cusp still remains in our simulation.Particles faster than density waves push them, and lose their kinetic energy.We have to calculate such power, but it needs non-linear analysis. It is quite difficult.Actually, for the run of T=10td, many Fourier component have been observed.The interactions between different Fourier modes are important in such cases.

Number of Cycles W/(dK/dt)From the argument of energy transfer rate, we notice that number of cycles is an important point to determine the density profile of DM halos.Our model predicts after 50 cycles, core scale will glow double compared with that of after 10 cycle, black arrow.To check it, we have performed N-body simulation.As expected, core scale doubles.However, it will be saturated because the number of cycles to exceed potential energy diverges around there.Thus we conclude the oscillation period, T determines the resultant core scale.

Small amplitudeEvolutionFrom the argument of energy transfer rate, a question arises.When the number of cycles is sufficiently large, Cusp-Core transition occur even if the amplitude of potential change is small, or not?To examine it, we have performed a N-body simulation adopting these parameters.This figure demonstrates that core creation occurs from the position of resonance of the fundamental tone, about 0.1 kpc.Effects of oscillation propagate to the innermost region.This result indicates even if the amplitude of potential change is small, Cusp-Core transition will occur when number of cycles are sufficient.I have to mention that this resultant core scale is quite similar to that of the run of large amplitude.Core scale is determined by T.GPUN-body SimulationsAim: Whether we can resolve the Core-Cusp problem or notDynamical evolution of DM halosGravity from whole of the system is dominant

Method: N-body simulations utilizing the Tree algorithm (Barnes & Hut 1986)Sufficient accuracyReasonable computational costManageable (no boundary, scale-free)Commonly used in computational astrophysics

The aim of our research is to investigate whether we can resolve the Core-Cusp problem in CDM context or not.So, we should follow the dynamical evolution of DM halos.DM halo is a kind of collisionless system in which gravity from whole of the system is essential.And so we choose N-body simulations utilizing the Tree algorithm as the numerical scheme.It keeps sufficient accuracy, and its computational cost is reasonable.Furthermore, this scheme is manageable.So it is commonly used in computational astrophysics.Tree Method (Barnes & Hut 1986)-particle(interact to i-particle)i-particle(compute gravity)(N: number of particles)

Next, I describe about the tree method briefly.It is an efficient algorithm to compute gravity among particles.Hereafter, we call particles computed gravity i-particles, and call particles interact to i-particle j-particle.The most straightforward and accurate scheme is the direct method.Thus the computational cost is proportional to N^2 because we compute gravity between all pairs of i-particle and j-particle. .N means number of particles.In the tree method, we treat sufficient far j-particles as a heavy j-particle.So we reduce number of net j-particles.Tree-Construction & Tree-TraversalTree-ConstructionSeparate cube (cell) contains all particles recursivelyLink to child- and brother-cellTree-Traversali-particles walk treeNear cell -> childFar cell -> brotherReduce computation

Tree algorithm consists of 2 parts, tree-construction and tree-traversal.In the first section, tree-construction, we construct oct-tree structure of particles.We separate cube contains all particles recursively. Hereafter, we call this cube cell.And we link all cells to child- and brother-cell.In the second section, tree-traversal, we make i-particles walk the tree structure constructed in the previous section.When i-particle reaches a near cell, it walks to child-cell. And if i-particle reaches a sufficient far cell, it walks to brother cell.In that case, j-particles contained in the cell are treated as a heavy j-particle. So we can reduce the amount of computation.

Overview of Our Tree-CodeSpeed-up a Tree-code by using GPU clusterSerial: Nakasato (2012)CPU: Tree-ConstructionGPU: Tree-TraversalParticle data: sorted by obeying space-filling curve Parallelization: MPIEach MPI process Is host of a GPUHas data of (N/number of MPI processes) i-particlesConstructs partial tree composed by its i-particlesCommunicates necessary tree data (Warren & Salmon 1993)

We speed-up a tree code by using GPU cluster.In the wake of Nakasato (2012), we make CPU cores compute tree-construction, and make GPU compute tree-traversal.And we sort particle data by obeying the Peano-Hilbert curve in the memory space to gain cache hit rate.Furthermore, we have parallelized our code by using MPI.Each MPI process becomes host of a GPU, and has partial particle data.It constructs partial tree composed by its i-particles, and communicates necessary tree data with other processes.

Warp Branch32 GPU cores (Warp) work concurrently (SIMD)Several kinds of processing in a warp (Warp branch)-> All instructions should be executed (Large overhead!)Warp branches in tree-traversal Occur whether each i-particle walks to child or brotherBecome less frequent by consolidating routes of i-particles in tree

Warp branch is an important issue for GPU computing. Hereafter, we explain the problem with NVIDIA Fermi architecture. In this architecture of GPUs, 32 cores work concurrently with the same instruction stream, in other words, GPU cores work in the manner of SIMD.This cluster of 32 cores called Warp.When several kinds of processing emergent in a warp, large overhead arises since all instructions should be executed.Warp branch in tree-traversal section occurs whether each i-particle walks to child or brother cell.It will become less frequent by consolidating routes of i-particles in tree.Vectorization & Grouping VectorizationConsolidating routes of i-particles in a GPU coreGroupingConsolidating routes of i-particles among some coresConsolidatingDecrease frequency of Warp branchIncrease the amount of computationOptimal value of vector length (V) and group member (G)Cell contains j-particlesGPU core 1GPU core 2To realize it, we propose two devisals.Step 1 is vectorization. We consolidate routes of i-particles in a GPU core by measuring distance to the cell from nearest particle in the core and judging the next step, brother or child, by using this value of distance.Step 2 is grouping. We consolidate routes of i-particles among several cores in the same fashion.Consolidating will decrease frequency of Warp branch, however it increases the amount of computation since it becomes hard to judge that the cell is sufficiently far from the i-particle.Thus it is supposed to exist optimal pair value of vector length and group member.

34Vectorization & Grouping VectorizationConsolidating routes of i-particles in a GPU coreGroupingConsolidating routes of i-particles among some coresConsolidatingDecrease frequency of Warp branchIncrease the amount of computationOptimal value of vector length (V) and group member (G)Cell contains j-particles35

HA-PACSGPU cluster installed at CCS of U-Tsukuba in 2012Base cluster: 268 nodes4 Tesla M2090 cards / node

We have implemented vectorization and grouping.We measure the performance of our code on the GPU cluster, HA-PACS.It has installed at CCS of U-Tsukuba in 2012.HA-PACS has 4 NVIDIA Tesla M2090 GPU cards per node.The total peak performance is about 800 TFLOPS.ImprovementComputation time of Tree-TraversalN=8M, single precisionSerial

time [sec]Number of group membersVector lengthCompared with the case of without Vectorization and Grouping ,

Tree-Traversal becomes more than 3 times faster!!

Optimal pair (V,G)=(4,4) The result of performance measurement is shown in this figure.We measured the computation time of Tree-Traversal.Compared with the case of without vectorization and grouping, sky blue circle, tree-traversal becomes more than 3 times faster.As expected, optimal pair value of vector length and number of group member exists.

Strong Scaling (N=32M)Time [sec]Number of GPUs 1/Np This figure shows the performance for strong scaling. We set the total number of particle, N = 32 millions. When Np is less than 10, the scaling is good, however, the performance saturates in the case of Np > 10. We guess the reason for performance saturation of kernel function (blue line) is inbarance of amount of computation among processes. Because we have assigned sorted (N/Np) particles to each process, some processes are assigned central part of particles while others are assigned outskirt part of them, where the number of cell traversal oneach process differs. We should change the way for particle assignment. The time to make LET (magenta) is nearly constant, and it is one of the dominant parts when Np > 10. In the current implementation, some CPU cores doesnt work. It is required to utilize them efficiently in order to accelerate the operation of making LET.

Weak Scaling (N=16M/GPU) Time [sec]Number of GPUs Log(Np) We also have shown the performance for weak scaling in this figure Each process holds N1= 16M particles. The total number of particles can be written by N = N1 Np. The time to compute gravity (red) is proportional to log Np, since the depth of the tree structure is proportional to log N log Np.Henceforth, we will overlap the computaion of kernel function with the making LTE and MPI communication to improve the performance.Summary -Tree Code-Tree code on GPU clustersVectorization & GroupingTo reduce the frequency of warp branchConsolidating the route of several i-particlesKernel function (tree traversal) Becomes more than 3 times fasterParallelization by using MPI

Future: Tree-SPH code

We are accelerating a tree code by using GPU clusters.Considering the features of GPUs, to reduce the frequency of warp branch is an important issue.To realize it, we propose two devisals: vectorization and grouping. We consolidate the route of i-particles in the tree structure.By implementing them, the kernel function, tree traversal part becomes more than 3 times faster.After some improvement, we will implement SPH code coupled with this tree code.