Script to collect AH shape information into one HDF5 file

Create issue
Issue #1828 closed
Frank Löffler created an issue

Attached is a python script which collects AH shape information and stores it into an HDF5 file. It also creates an xml file (xdmf), such that visit can open it without any special reader.

The script itself works for python 2 and 3, and requires h5py to be present.

It will search for all horizon shape files within the current directory, and consider all of them for inclusion. This could be turned into a command line option: feel free.

Currently, the script requires one argument: a valid Cactus HDF5 file, intended to be from the same run and containing all time steps the horizon should be visualized in. The combined AH file(s) will contain the same time steps that are present in the input HDF5 file. If no horizons were found at a certain iteration, a dummy grid will be inserted - such that visit automatically merges time sliders even if horizons are not found at every output iteration.

This script should probably go someplace within the AHFinderDirect thorn, as utility. Please suggest a better name if you don't like the current one.

Keyword:

Comments (9)

  1. Roland Haas
    • removed comment

    Looks fine to me in general. Comments:

    • the name should not contain "visit" since xdmf files are also readable by other tools eg Paraview, instead maybe "CollectAHShapesInXdmf.py"
    • why is there a need to specify an HDF5 file? I would sometimes like to just visualize the horizon without having any hdf5 output present (eg for binary black hole simulations, see eg https://www.youtube.com/watch?v=ZsODZW0VuhQ)
    • the script makes assumptions on the name of the files which are not always true since the AHFinderDirect parameter h_base_file_name lets you change the "h" to something else (see comment in param.ccl). It should document these assumptions more prominently rather than only in the middle part of the source code.
    • would it be possible to also support AHFinderDirect's output_HDF5_files option?
    • an alternative to having Xdmf+HDF5 may be to store all data inline in the Xdmf file (ASCII in that case) if this is not provibitively large. It would have the advantage of fewer files having to be carried around
    • the script should support a --help option explicitly, the help text should start with the word "Usage", required argument should not be enclosed in brackets but be bare, the fount of all knowledge suggests a particular format.
    • it would be nice if error message (the usage is no error message) were output to stderr rather than stdout
    • rather than adding dummy meshes, would it not be better to re-use the mesh from the last timestep? This would be visually less jarring than having an AH vanish
    • this is an if(True) statement in there. We should either do as the comment above says or remove the "else" branch. Users should not have to modify the script code to chose options.

    • compare this to Eloisa's older scripts http://svn.cactuscode.org/VizTools/VTKutils/trunk/AHFVTK/ which produce VTK files instead (they require fewer tools, only bash and (g)awk).

  2. Ian Hinder
    • removed comment

    Thanks for the script!

    How hard would it be to modify AHFinderDirect to output such files directly? Could it be done already in a parameter file by storing the AH shape in a spherical surface and then outputting the spherical surface using CarpetIOHDF5? This information would be quite "raw". AHFinderDirect could then maybe output a corresponding XMDF file so it would be easy to open in a visualisation tool.

    I agree with Roland that it would be really good if the script could be used without a reference HDF5 file. For example, maybe you could provide a list of times or iterations in an argument, or another file, or in standard input. Or maybe just arguments for --starttime, --endtime, --deltatime. Or the corresponding iteration equivalent.

    Is there a place that such scripts are all listed?

  3. Frank Löffler reporter
    • removed comment

    Thanks for your reviews. There are obviously many ways to improve this script. Some I implemented now, some others are invited to.

    Replying to [comment:2 rhaas]:

    • the name should not contain "visit" since xdmf files are also readable by other tools eg Paraview, instead maybe "CollectAHShapesInXdmf.py"

    You are right. Maybe just AH2xdmf.py?

    • why is there a need to specify an HDF5 file? I would sometimes like to just visualize the horizon without having any hdf5 output present (eg for binary black hole simulations, see eg https://www.youtube.com/watch?v=ZsODZW0VuhQ)

    Again, you are right. In the case I needed it for, I had to align time steps to existing hdf5 files. That argument is optional now, and if not given all timesteps in AHs are used.

    • the script makes assumptions on the name of the files which are not always true since the AHFinderDirect parameter h_base_file_name lets you change the "h" to something else (see comment in param.ccl). It should document these assumptions more prominently rather than only in the middle part of the source code.

    It now is part of the help text. It would be difficult to find files with non-default names though, unless one were to add an option for the regex which matches those files, and returns the horizon and timestep numbers. This is possible, and such a contribution would be welcome.

    • would it be possible to also support AHFinderDirect's output_HDF5_files option?

    I have to confess, I didn't even know about that option. Of course that would be most desirable, and maybe then one wouldn't even need to create a separate h5 file but could simply use the existing one. Since I don't have any of such output right now, however, I will only add this later, or someone else is welcome now.

    • an alternative to having Xdmf+HDF5 may be to store all data inline in the Xdmf file (ASCII in that case) if this is not provibitively large. It would have the advantage of fewer files having to be carried around

    One could, but I didn't try. A typical horizon shape needs on the order of 50kB if stored in binary format (obviously depending on N_rho and N_theta). hdf5 adds overhead to that. hdf5 compression isn't efficient for that size too. I would estimate ASCII to need a small factor (2-3?) more than that, which wouldn't make it a size issue. For an example of about 3400 horizons (19x19), I get an hdf5 file size of about 240MB. Reading ASCII is most likely much slower, having to parse the ASCII values, but with that size that should also not be a problem. In short: I don't see a reason why it shouldn't work, but I didn't see a reason why to go that route. I came "from the other side": I started with hdf5 and wanted an xml description of the file so a visualization tool can just read it. Your argument with the number of files is a good one, but reducing the gazillion AH files (for each timestep) to just two per horizon is already sufficient for me.

    • the script should support a --help option explicitly, the help text should start with the word "Usage", required argument should not be enclosed in brackets but be bare, the fount of all knowledge suggests a particular format.

    It now has. Thanks for that. It doesn't use a full-blown option parser though (yet).

    • it would be nice if error message (the usage is no error message) were output to stderr rather than stdout

    It now does.

    • rather than adding dummy meshes, would it not be better to re-use the mesh from the last timestep? This would be visually less jarring than having an AH vanish

    Not if you have an AH disappear, like you can have in BBH simulations. One could add an option for this, which most likely should be per horizon. Once the script has found a home everyone is welcome to add that possibility.

    • this is an if(True) statement in there. We should either do as the comment above says or remove the "else" branch. Users should not have to modify the script code to chose options.

    I agree. I just didn't have the patience to fully implement possible options, but had the code for one obvious case already there. It would have been waste to just delete it, so I "commented" it that way. At least it would be an easy code change. Having said that - now that the input hdf5 file is optional, both code paths can actually be taken.

    I looked into this and even used it once or twice (Thanks Elo!), but if I remember correctly, vtk doesn't (easily) support multiple timesteps per file (or visit's reader didn't) - and once I had to 'roll my own' anyway I felt that python would be a better choice for parsing text files and writing data to hdf5.

    Replying to [comment:3 hinder]:

    How hard would it be to modify AHFinderDirect to output such files directly?

    It shouldn't be very hard. In my case that was not an option. I already had the data and didn't want to re-run. Also, things would get complicated when multiple restarts come into the game.

    Could it be done already in a parameter file by storing the AH shape in a spherical surface and then outputting the spherical surface using CarpetIOHDF5?

    Outputting the spherical surface would be a good option (also because other thorns would benefit from that too). I was thinking about that but then found the AH files easy to parse and the fastest route to what I needed. A similar script for SphericalSurface output shouldn't be that complicated.

    Is there a place that such scripts are all listed?

    We do have a 'tools'/'utilities' repository, but in that case I thought about putting it into the AHFinderDirect repository, since it is obviously only reading those output files and would be useless without.

  4. Log in to comment