Inclusion of FUKA importer thorns

[tootle]

The FUKA importer thorns currently consists of a base thorn that handles dependencies and the related Importer thorn that interfaces between the Einstein Toolkit and the FUKA exporters that are built into the library. This allows for import of FUKA generated initial data for isolated black holes and neutron stars as well as binary black hole, binary neutron stars, and black hole-neutron star binaries.

The prototype thorns which have been used extensively for multiple published works are available at https://bitbucket.org/fukaws/

However, a revised importer will be written that meets the standards for inclusion as an official thorn.

[Roland Haas]

@Konrad Topolski will review.

‌

[tootle]

@Roland Haas @Konrad Topolski
I have put some effort into the KadathThorn which handles ensuring the kadath library exists and, if not, will be built along with all the other thorns. This still requires that the user has a compiler that is compatible with the C++17 standard and std::filesystem.

I have added quite some output during the build process to let the user know that HOME_KADATH will need to be set in their submit script along with the directory to use. I’m not sure if there is a good way to include this in the basic simfactory files? Maybe @Roland Haas has a good idea for this since I never use simfactory. It is worth noting that this isn’t critical other than to utilize ${HOME_KADATH}/eos for using the built-in EOS files for 1D tables and polytropes.

@Roland Haas : Additionally, I would prefer to avoid having a tarball of the library that is separately packaged and stored as an ExternalLibrary. To avoid this - during the build procedure for KadathThorn (if FUKA needs to be built) - the build script will pull the fuka submodule. I’m not sure if this goes against any best practices that ETK tries to adhere by, but it worked quite well from a testing/build standpoint.

@Konrad Topolski : The thorns are more or less in a final place, but I still need to write the documentation. I don’t think this will impact you though given your familiarity with the codes. The thing I would note is that in the KadathImporter there is now a test directory that includes a quick test using preexisting BH initial data. The only output is from AHFinder where one can see the BH mass matches the ID in the info file. I have also included a semi-minimum thornlist needed to run the test along with the cfg and sub scripts that I used to build ETK and submit the test job. The build procedure and test was performed on my laptop. The test, using 4 ranks, took ~2min.

Cheers

P.S. The latest of both thorns can be obtained by adding the following to a standard thornlist:

# fuka thorns
!TARGET   = $ARR
!TYPE     = git
!URL      = https://bitbucket.org/fukaws/$2
!REPO_PATH=../$2
!CHECKOUT =
Fuka/KadathImporter
#disabled fuka/kadathpizza
Fuka/KadathThorn

‌

[tootle]

@Konrad Topolski The initial draft documentation is now also there.

[Konrad Topolski]

@tootle The GenericFD should most likely be removed from the bhtest.par given the provided minimal thornlist.

In documentation.tex of , the relative path should read:
\usepackage{../../../../doc/latex/cactus}
in order for the make KadathThorn-ThornDoc invoked in /Cactus
to produce a pdf file.
On the other hand, if this is done, the manual compilation of documentation.tex via pdflatex documentation.tex breaks.

I have looked at the SphericalSurface thorn, which is similarly placed in a parent directory CactusNumerical and uses \usepackage{../../../../doc/latex/cactus}; for some reason no problems occur there given either method of generating the documentation.

Apart from that no objections to the content of the code, tests or documentation.

@tootle , @Roland Haas
I have finished the review following the relevant guidelines.

[tootle]

@Konrad Topolski

Thanks for the feedback. I will make the correction to the par file. Regarding the documentation this is likely because the Thorns are stored in arrangement/FUKA/<thorn name> but the repos are stored in repos/<thorn name> which is not the case for SphericalSurfaces. I’ll see if I can fix this and will ask you to double check before I say that everything is “done”

[tootle]

The documentation issue is not easily resolved due to the differences in relative path. This issue doesn’t exist for spherical surfaces as I mentioned before due to the same relative paths in the arrangement and repos directory. This is due to the fact that SphericalSurfaces is folder in the cactusnumerics repo whereas, kadaththorn and kadathimporter are their own repositories. No changes will be made regarding this point.

[Konrad Topolski]

Along with these changes, all concerns have been addressed
and the review is complete.

‌

[tootle]

The latest changes have been pushed to resolve the documentation issue such that make KadathThorn-ThornDoc will work consistently as with other thorns.

Small revision was made to the build script to allow different base foldernames in case someone pulls the thorn under a different foldername. Tested to verify compilation still functions as expected

@Samuel Cupp @Roland Haas : I will consider this complete for now until I hear further

[Samuel Cupp]

Great! Thanks for the quick turnaround on both the review and the requested changes. As for your email question regarding adding the thorn to the master thornlist, I recommend making a PR on the EinsteinToolkit repo.

[Roland Haas]

The current thornlist contains:

#DISABLED fuka/kadathpizza # Only for use with WhiskyTHC + PizzaBase

but

(a) https://bitbucket.org/fukaws/kadathpizza is private (I get a 404 error trying to access)

(b) Pizza is not part of the eT

(c) capitalization matters, fuka and Fuka are different arrangements

I would suggest to remove the line from the thornlist (GetComponents will fail with a failed download).

[tootle]

@Roland Haas No issue with removal.

As an FYI for those they may be interested, this is simply a typo and the line should read:
#DISABLED Fuka/kadath_pizza # Only for use with WhiskyTHC + PizzaBase

[Roland Haas]

That still leaves point (b). Note that if kadath_pizza (2 typos actually, “F != f” and “katathpizza != kadath_pizza”) is included then, to match what is done for the other DISABLED thorns (which can be enabled using simfactory’s enable-thorn machine ini option) then kadath_pizza should show up in the !CHECKOUT line so that GetComponents checks it out, but Cactus does not use it. See eg ReprimAnd.

[Roland Haas]

With the inclusion progressing, would it be possible to add push hooks so that commit emails are sent to the ET commits mailing list? See here: https://docs.einsteintoolkit.org/et-docs/Services#git

[tootle]

@Roland Haas

Would you like a new pull request for the thornlist.

As I said, no issue with excluding this line since THC is not part of the toolkit. I have provided in the documentation details related to what needs to be handled by the EOS manager (or by a third party thorn such as kadath_pizza).

I will look into the push hooks you mentioned as I'm not familiar with this.

[Roland Haas]

Including KadathThorn does harm. It fails to compile with a missing boost include file with the default ET thornlist. See:
https://github.com/EinsteinToolkit/tests/actions/runs/4502539206/jobs/7924641336#step:9:17102

Note that Boost is not in the ET (and is hard to include by default since it takes a very long time to compile). Reprimand depends on it, but it is not compiled in by default and @Wolfgang Kastaun is, as far as I know, removing the dependency (does not work on GPUs).

[tootle]

Unfortunately this is not a dependency that can be easily removed so I will have to exclude the importers from the official release. If this changes I will consider inclusion at a later date.

[Roland Haas]

Is it possible to include the (hopefully small) parts of Boost needed in the KadathThorn source code?

While I can see that eventually Boost may be included in the ET (for Kadath, Reprimand, CarpetX), it is a large amount of code and not always good about backwards compatibility so one can likely end up a situation where different thorns require conflicting Boost versions.

[tootle]

For the importer, the only portions of Boost needed are those related to ptree, which handles the ingestion of the config file for FUKA initial data solutions. I can look into whether or not it is possible to include a subset of boost, but I don't want to hold up the release since it is already quite late in the release schedule.

As for including boost in the ETK, I can understand the apprehension as it can be exceptionally large and time consuming to build depending on how many of the components one needs to install. In our case, the systems we work on have boost installed by default so it was a natural dependency to leverage especially for python bindings. Unfortunately this is not always the case especially for older systems

[Roland Haas]

I see. An ET provided Boost ExternalLibrary would have to include all of Boost, both the STL template parts and the compiled parts, since we would not know what each thorn uses.

If the dependency persists the is should be listed in configuration.ccl along with GSL etc.

You may also need to consider an OPTIONAL dependency on CMake in particular if you require a non-ancient cmake. Some of CarpetX dependencies do that (AMReX, ADIOS I think) and can optionally use a cmake in ${CMAKE_DIR}/bin/cmake instead of just cmake. I think the two of them between them now require cmake 3.23 (from around July 2022, so very very recent for HPC clusters).

[Roland Haas]

I marked the thorns as DISABLED. We can include them like this for the next release an selectively enable them on clusters where we know Boos to be present (same as eg for PETSc based thorns). That is if the release managers are ok and the final vote in the ET call goes well.

[tootle]

@Roland Haas I had a look today and the dependencies that ptree relies within boost are too extensive to try and extract manually unfortunately. Therefore, the only option in the short term would be to include it as DISABLED if the ETK maintainers agree or to exclude these thorns altogether.

If the dependency persists the is should be listed in configuration.ccl along with GSL etc.

This would still require there to be an intermediate thorn that PROVIDES BOOST.

You may also need to consider an OPTIONAL dependency on CMake in particular if you require a non-ancient cmake. Some of CarpetX dependencies do that (AMReX, ADIOS I think) and can optionally use a cmake in ${CMAKE_DIR}/bin/cmake instead of just cmake. I think the two of them between them now require cmake 3.23 (from around July 2022, so very very recent for HPC clusters).

This is an interesting point. In reality the current version of KADATH is based on cmake 2.8 which I’ve recently had to start manually removing old safeties that interfere with methods implemented in cmake >3.2. Specifically, base KADATH included its on modules for find_package for its dependencies which would override the modern default scripts. This meant that all the search routines for looking for packages in $ENV{<PACKAGE NAME>_ROOT} were not searched since this was not implemented until cmake 3 I believe. I know there are those that unfortunately have to deal ith systems running old libraries, but this is not something I deal with at the moment so it is difficult to test and develop against.

[Wolfgang Kastaun]

@tootle Is the disabled kadath_pizza part simply using the EOS framework from WhiskyTHC? Because the RePrimAnd thorn that is part of the ET includes this functionality and the interface is very similar (I was the one who wrote the “pizza” parts of WhiskyTHC). So maybe it would make sense to just use RePrimAnd instead.

[Roland Haas]

I see.

Yes, an intermediate thorn (an ExternalLibrary) that provides Boost would be needed (right now, for CarpetX we use David Radice’s thorn: https://github.com/dradice/Boost, which provides a built in copy of Boost 1.55 or interfaces with a system installed Boost. It uses an older style of building things, than eg is used by “modern” ExternalLibraries like HDF5 or GSL, but actually that happens to work well for Boost).

Even if Boost is not part of the ET KadathThorn should declare a dependency on Boost in its configuration.ccl (same as the Reprimand helper thorn does), since this way a non-system Boost can be interfaced correctly (Cactus sets the relevant -I and -L options).

CMake is a mixed bag on clusters usually. It is very much geared towards workstation setups where there is exactly one copy of a library and its find_XXX functions seem to not always follow a set pattern or let a user override settings (autoconf is similar,but usually at least lets one override everything with CFLAGS and LDFLAGS which CMake resists). So I expect issues :-)

[tootle]

@Wolfgang Kastaun Thank you for your comment. Indeed, I’m aware of your work and have looked at the new RePrimAnd library a bit, however, I haven’t worked with the new thorn nor looked at what would be needed to interface the Fuka thorns with another EOS framework.

The kadath_pizza thorn is quite trivial, but plays a necessary role. For simulations that utilize a tabulated equation of state, the initial temperature and Ye need to be initialized prior to the ID_CONVERT stages. For our internal codes, the EOS manager initializes these values for us based on the extracted cold beta equilibrium slice, however, this was not the case when I was trying to run some comparison tests with WhiskyTHC. Instead of making the importer thorn more bloated I decided to write kadath_pizza to keep this functionality isolated and modular since interfaces differ between EOS frameworks.