This pull request, which is not yet ready to go in, includes all of the modifications necessary to load in very large SDF output files and analyze them, including HTTP support. This is part of a larger effort, which is ongoing.
I have opened this pull request to keep track of the modifications I need to make to this PR before it can be finalized and merged in. Already I know about:
Overall style cleanups, as necessary
Figuring out the thingking/httparray dependencies (@samskillman, @mswarren)
Resolve any conflicts
Adding tests for the SDF frontend
UPDATE: I think this is now ready for review.
UPDATE: This is now up-to-date with the current state of the yt-3.0 repository, and the thingking imports have been sanitized. I'd appreciate review, comments, etc, as I would like this to go in very soon.
It looks like those changes never got backed out - was that on purpose?
This PR isn't really ready for comments yet; I opened it (as I noted above) as a simple way of keeping track of modifications, which is why only the other two contributors were listed as reviewers. Comments are welcomed, but they won't really be responded to until after the checklist of items is done.
We could make use of install_requires (see here and elsewhere in the setuptools docs) in our setup.py script so that thingking (as well as our other pip-installable dependencies) get installed if they cannot be imported.
Does thingking need to be unconditionally imported for the SDF frontend to work? It looks like it's only used in a few places.
Perhaps you can import it inline in the HTTPDataStruct and HTTPSDFRead inside of a try/except block and report an error message about needing thingking for that functionality. You could take a look at how we handle astropy imports to see one possible solution.