HTTPS SSH - Pre-sort dark matter particles for nested cosmology Enzo

Quick Intro:

When running a nested cosmology simulation in Enzo, the procedure is typically
as follows (assuming one is using Parallel IO, which is generally advisable):
1. Run inits.exe two or more times creating equal numbers of initial condition
files describing dark matter particle and possibly baryon information.
2. Rung ring.exe on the particle files, splitting them up into an equal number
of files to the number of tasks that will be used to run enzo.exe.
3. Run enzo.exe.

This script is run between steps 1 and 2, and it produces a corrected set
of particle initial condition files.

*** Note: *** using this method, one must produce a particle mass file with
inits.exe, and read the particle masses in when starting up enzo.exe.


This requires an install of Python, along with the 'h5py' and 'numpy' modules.
The simplest way to gain an installation with these capabilities is to install
a copy of yt, which is a powerful community tool for analyzing AMR grid-based
data, such as enzo. Please visit

Detailed Procedure:

1. When setting up the inits.exe parameter files, specify a particle mass file
using a line like this:

ParticleMassName     = ParticleMasses.0

The suffix (e.g. .0) should match the other fields specified in that inits file.

Run inits.exe for nested initial conditions as normal.

2. Place a copy of in the same directory as the generated initial
conditions. Edit the settings at the top of the file to match your simulation.
Please note that the boundaries used for the (sub)grids should only be the
*outer* edges, so even if there are several subgrids with inner and outer 
boundaries, specify *only* the outer boundaries.

The script relies on Python modules that use compiled C code, so the runtime
should be mainly limited by disk speed. Of course, the data for the initial
conditions needs to fit into memory. Future improvements may include a
'chunking' mode which would allow fixing unlimited size datasets.

3. Run the script:

$ python

The corrected initial condition files are placed in a new directory, named
'new_ICs'. Please check that the new files are correct by checking
the total particle count. It is not advisable to delete the original initial
condition files. Move the old to a safe place, and move the corrected ones
in their place.

4. Run ring.exe and enzo.exe using the corrected files.

Problem Explanation:

Inits.exe creates the positions for the dark matter particles based upon the
velocities of the particles. The calculation puts no absolute limit on the
positions, beyond reason, of course.

For particles placed near the periodic boundaries, particles that extend past
the boundary are reflected to the opposite side of the box. However, for
particles created for nested subgrids, a particle being placed outside that
subgrid presents several problems. First, when the initial conditions are
split by ring.exe, it sees a subgrid particle being outside the subgrid as an
error, and throws them out. The result appears to be a loss of some fraction
of the particles. Second, because Enzo represents particle masses using
densities, when particles shift in Enzo between refinement levels, their mass
value changes. To account for this, this script keeps track of the direction
particles move and adjusts the mass to compensate for the shift. This is why
the ParticleMass file must be created by inits.

Further Information:

As always, please visit the LCA website for more information, the mailing list
and software updates.


This README and was written by Stephen Skory,

December, 2009