SDF frontend and Dark Sky Sims work

#955 Merged at 990c27c
  1. MattT

This pull request, which is not yet ready to go in, includes all of the modifications necessary to load in very large SDF output files and analyze them, including HTTP support. This is part of a larger effort, which is ongoing.

I have opened this pull request to keep track of the modifications I need to make to this PR before it can be finalized and merged in. Already I know about:

  • Overall style cleanups, as necessary
  • Figuring out the thingking/httparray dependencies (@samskillman, @mswarren)
  • Resolve any conflicts
  • Adding tests for the SDF frontend

UPDATE: I think this is now ready for review.

UPDATE: This is now up-to-date with the current state of the yt-3.0 repository, and the thingking imports have been sanitized. I'd appreciate review, comments, etc, as I would like this to go in very soon.

UPDATE: Passes tests.

UPDATE: All comments addressed!

Comments (32)

  1. MattT author

    This PR isn't really ready for comments yet; I opened it (as I noted above) as a simple way of keeping track of modifications, which is why only the other two contributors were listed as reviewers. Comments are welcomed, but they won't really be responded to until after the checklist of items is done.

  2. MattT author

    This PR will also require an update to Rockstar, but fortunately will bring it back in line with the mainline Rockstar.

  3. Nathan Goldbaum

    Looks like the server yt-fido runs on is having issues at the moment. Hopefully testing will be back up tomorrow.

  4. Nathan Goldbaum

    We could make use of install_requires (see here and elsewhere in the setuptools docs) in our script so that thingking (as well as our other pip-installable dependencies) get installed if they cannot be imported.

  5. Nathan Goldbaum

    Does thingking need to be unconditionally imported for the SDF frontend to work? It looks like it's only used in a few places.

    Perhaps you can import it inline in the HTTPDataStruct and HTTPSDFRead inside of a try/except block and report an error message about needing thingking for that functionality. You could take a look at how we handle astropy imports to see one possible solution.

  6. MattT author

    Looks to me like the issues for the tests are related to LeftEdge/RightEdge not being in code units anymore. I can replicate them here.

    1. MattT author

      Help on figuring out why the changes to selection_routines broke it would be very helpful. Seems to me like ensure_code didn't cause the breakage.