#1384 Merged at 6aa1f1d
Repository
MatthewTurk
Branch
yt
Repository
yt_analysis
Branch
yt
Author
  1. MattT
Reviewers
Description

(Commit e88d27c has more details)

This implements a new scheme for potentially fast indexing of patch-based AMR data. It is not yet faster, and it is currently disabled. However, it moves a considerable amount of computation into Cython, removes almost all of the caching of child masks and selector masks from grid patch objects (for what should be an extremely large memory savings on big patch datasets) and should when implemented result in a much bigger performance boost for things like projections, ghost zones, and the like.

There are several caveats:

  • It is currently disabled
  • It uses the existing grid patch object API as a fallback
  • It will not (yet) work with datasets where a child patch has non-unique parentage; this should be a simple fix but it is not yet implemented.
  • refinement factor is exclusively 2, but can be changed.
  • As noted above, it is not yet faster. For the small dataset I have been testing with, it's roughly the same speed as the old method, but importantly it does not rely on any caching of selection or child masking. Both of these could be huge speedups, but as-is the memory win is pleasing. (i.e., no more uint8_t masks of selection and children for every grid in a selector object.)

More on this as time permits. I am optimistic that this will eventually replace the existing methods for selecting data in patch based AMR for non-"grid" data selection.

  • UPDATE 1: This is ready for review. As it stands, behavior is unchanged. But it lays the groundwork for future fast indexing systems, so I would like to get it landed as a staging ground before building it further and enabling it.

Comments (13)

  1. chummels

    As someone who has never touched this part of the code, I really don't know what is going on here. Would it be possible to include some comments and docstrings on major functions, or even a basic description of what is going on, so that I (we) can best review the code? Other than that, I'd just be looking for small coding mistakes.

    1. Britton Smith

      I have to agree that the Cython code in particular could use some comments or docstrings on the functions to explain what's happening.

      On a separate note, I see that there are unit tests added, but are there also answer tests in place that will fail if this has changed getting field data from containers?

      1. MattT author

        Yup, answer tests should fail if the data changes -- but it's still turned off. I'll add comments to the Cython code, but I am not yet sure about adding docstrings since most/all of them are private methods.

        One note is that the grid_tree stuff is all from @John ZuHone , with minor cleanups.