Reduce overall memory usage and set Enzo particle_mass to be mass always

#55 Merged at 67d98f2
Repository
MatthewTurk
Branch
yt-3.0
Repository
yt_analysis
Branch
yt-3.0
Author
  1. MattT
Reviewers
Description

This turns on/off caching of properties such as icoords, ires, fcoords, and fwidth. The same benchmarks were run as in pull request 54: https://bitbucket.org/yt_analysis/yt-3.0/pull-request/54/octree-diet and here are the results.

           enzo: 2.323e+16 Msun within 1 Mpc (9.199e-01 GB added, took 3.073e+02 seconds)
         gadget: 4.241e+11 Msun within 1 Mpc (2.754e-01 GB added, took 2.154e+01 seconds)
       gasoline: 8.320e+10 Msun within 1 Mpc (-2.051e-02 GB added, took 4.159e+01 seconds)
        pkdgrav: 6.179e+11 Msun within 1 Mpc (2.305e-01 GB added, took 1.007e+02 seconds)
         ramses: 4.303e+11 Msun within 1 Mpc (5.664e-02 GB added, took 3.805e+01 seconds)
Total memory usage:   1.521e+00 GB

Overall, the total memory usage is 2 gigabytes smaller, all from Enzo. I believe that the same memory reduction will be present in any other patch-based code.

Note that each of the frontends sees a slowdown, but overall the memory usage has gone down considerably. The remaining gains of memory in Enzo seem (to my eyes) to be related to caching of child_masks, but I have not yet determined if that is feasible to eliminate. I have left comments in each of the commits that directs attention to why decision were made at various times.

This also includes a change to how Enzo's particle_mass field works. It has been 100% special cased in the IO routines themselves. Enzo outputs a particle mass that has been pre-divided by the cell volume; we multiply by the cell volume in the IO handler, where we know the grid's size, to avoid costly and time-consuming double-iteration from a NeedsGridType validator. In doing so, we make it slightly harder to access the actual on-disk field (but not that much harder) and change what the field means in yt as opposed to Enzo. However, nearly always we actually want the mass to be a mass, not a density, so I believe this change is acceptable.

Comments (3)

  1. Ji-hoon Kim

    Tested on Enzo, Ramses, 2 versions of Tipsy, and 4 versions of Gadget datasets. Works great, and in particular, Enzo's particle fields are now read in properly.

  2. Nathan Goldbaum

    After Ji-hoon's comment I went and checked whether this helps with the active particle selection issue I mentioned on IRC. No dice, unfortunately.

    1. MattT author

      I think I have an idea for a generic solution to the problem of active particles. I will attempt to hotfix it in a separate PR, and then attempt to implement the generic solution; this solution will also work towards generic filtering. Seeing as how this doesn't cause any new problems, can it be accepted?