Inclusion of FUKA importer thorns

[tootle]

The FUKA importer thorns currently consists of a base thorn that handles dependencies and the related Importer thorn that interfaces between the Einstein Toolkit and the FUKA exporters that are built into the library. This allows for import of FUKA generated initial data for isolated black holes and neutron stars as well as binary black hole, binary neutron stars, and black hole-neutron star binaries.

The prototype thorns which have been used extensively for multiple published works are available at https://bitbucket.org/fukaws/

However, a revised importer will be written that meets the standards for inclusion as an official thorn.

[Roland Haas]

@Konrad Topolski will review.

‌

[tootle]

@Roland Haas @Konrad Topolski
I have put some effort into the KadathThorn which handles ensuring the kadath library exists and, if not, will be built along with all the other thorns. This still requires that the user has a compiler that is compatible with the C++17 standard and std::filesystem.

I have added quite some output during the build process to let the user know that HOME_KADATH will need to be set in their submit script along with the directory to use. I’m not sure if there is a good way to include this in the basic simfactory files? Maybe @Roland Haas has a good idea for this since I never use simfactory. It is worth noting that this isn’t critical other than to utilize ${HOME_KADATH}/eos for using the built-in EOS files for 1D tables and polytropes.

@Roland Haas : Additionally, I would prefer to avoid having a tarball of the library that is separately packaged and stored as an ExternalLibrary. To avoid this - during the build procedure for KadathThorn (if FUKA needs to be built) - the build script will pull the fuka submodule. I’m not sure if this goes against any best practices that ETK tries to adhere by, but it worked quite well from a testing/build standpoint.

@Konrad Topolski : The thorns are more or less in a final place, but I still need to write the documentation. I don’t think this will impact you though given your familiarity with the codes. The thing I would note is that in the KadathImporter there is now a test directory that includes a quick test using preexisting BH initial data. The only output is from AHFinder where one can see the BH mass matches the ID in the info file. I have also included a semi-minimum thornlist needed to run the test along with the cfg and sub scripts that I used to build ETK and submit the test job. The build procedure and test was performed on my laptop. The test, using 4 ranks, took ~2min.

Cheers

P.S. The latest of both thorns can be obtained by adding the following to a standard thornlist:

# fuka thorns
!TARGET   = $ARR
!TYPE     = git
!URL      = https://bitbucket.org/fukaws/$2
!REPO_PATH=../$2
!CHECKOUT =
Fuka/KadathImporter
#disabled fuka/kadathpizza
Fuka/KadathThorn

‌

[tootle]

@Konrad Topolski The initial draft documentation is now also there.

[Konrad Topolski]

@tootle The GenericFD should most likely be removed from the bhtest.par given the provided minimal thornlist.

In documentation.tex of , the relative path should read:
\usepackage{../../../../doc/latex/cactus}
in order for the make KadathThorn-ThornDoc invoked in /Cactus
to produce a pdf file.
On the other hand, if this is done, the manual compilation of documentation.tex via pdflatex documentation.tex breaks.

I have looked at the SphericalSurface thorn, which is similarly placed in a parent directory CactusNumerical and uses \usepackage{../../../../doc/latex/cactus}; for some reason no problems occur there given either method of generating the documentation.

Apart from that no objections to the content of the code, tests or documentation.

@tootle , @Roland Haas
I have finished the review following the relevant guidelines.

[tootle]

@Konrad Topolski

Thanks for the feedback. I will make the correction to the par file. Regarding the documentation this is likely because the Thorns are stored in arrangement/FUKA/<thorn name> but the repos are stored in repos/<thorn name> which is not the case for SphericalSurfaces. I’ll see if I can fix this and will ask you to double check before I say that everything is “done”

[tootle]

The documentation issue is not easily resolved due to the differences in relative path. This issue doesn’t exist for spherical surfaces as I mentioned before due to the same relative paths in the arrangement and repos directory. This is due to the fact that SphericalSurfaces is folder in the cactusnumerics repo whereas, kadaththorn and kadathimporter are their own repositories. No changes will be made regarding this point.

[Konrad Topolski]

Along with these changes, all concerns have been addressed
and the review is complete.

‌

[tootle]

The latest changes have been pushed to resolve the documentation issue such that make KadathThorn-ThornDoc will work consistently as with other thorns.

Small revision was made to the build script to allow different base foldernames in case someone pulls the thorn under a different foldername. Tested to verify compilation still functions as expected

@Samuel Cupp @Roland Haas : I will consider this complete for now until I hear further

[Samuel Cupp]

Great! Thanks for the quick turnaround on both the review and the requested changes. As for your email question regarding adding the thorn to the master thornlist, I recommend making a PR on the EinsteinToolkit repo.

[Roland Haas]

The current thornlist contains:

#DISABLED fuka/kadathpizza # Only for use with WhiskyTHC + PizzaBase

but

(a) https://bitbucket.org/fukaws/kadathpizza is private (I get a 404 error trying to access)

(b) Pizza is not part of the eT

(c) capitalization matters, fuka and Fuka are different arrangements

I would suggest to remove the line from the thornlist (GetComponents will fail with a failed download).

[tootle]

@Roland Haas No issue with removal.

As an FYI for those they may be interested, this is simply a typo and the line should read:
#DISABLED Fuka/kadath_pizza # Only for use with WhiskyTHC + PizzaBase

[Roland Haas]

That still leaves point (b). Note that if kadath_pizza (2 typos actually, “F != f” and “katathpizza != kadath_pizza”) is included then, to match what is done for the other DISABLED thorns (which can be enabled using simfactory’s enable-thorn machine ini option) then kadath_pizza should show up in the !CHECKOUT line so that GetComponents checks it out, but Cactus does not use it. See eg ReprimAnd.

[Roland Haas]

With the inclusion progressing, would it be possible to add push hooks so that commit emails are sent to the ET commits mailing list? See here: https://docs.einsteintoolkit.org/et-docs/Services#git

[tootle]

@Roland Haas

Would you like a new pull request for the thornlist.

As I said, no issue with excluding this line since THC is not part of the toolkit. I have provided in the documentation details related to what needs to be handled by the EOS manager (or by a third party thorn such as kadath_pizza).

I will look into the push hooks you mentioned as I'm not familiar with this.

[Roland Haas]

Including KadathThorn does harm. It fails to compile with a missing boost include file with the default ET thornlist. See:
https://github.com/EinsteinToolkit/tests/actions/runs/4502539206/jobs/7924641336#step:9:17102

Note that Boost is not in the ET (and is hard to include by default since it takes a very long time to compile). Reprimand depends on it, but it is not compiled in by default and @Wolfgang Kastaun is, as far as I know, removing the dependency (does not work on GPUs).

[tootle]

Unfortunately this is not a dependency that can be easily removed so I will have to exclude the importers from the official release. If this changes I will consider inclusion at a later date.

[Roland Haas]

Is it possible to include the (hopefully small) parts of Boost needed in the KadathThorn source code?

While I can see that eventually Boost may be included in the ET (for Kadath, Reprimand, CarpetX), it is a large amount of code and not always good about backwards compatibility so one can likely end up a situation where different thorns require conflicting Boost versions.

[tootle]

For the importer, the only portions of Boost needed are those related to ptree, which handles the ingestion of the config file for FUKA initial data solutions. I can look into whether or not it is possible to include a subset of boost, but I don't want to hold up the release since it is already quite late in the release schedule.

As for including boost in the ETK, I can understand the apprehension as it can be exceptionally large and time consuming to build depending on how many of the components one needs to install. In our case, the systems we work on have boost installed by default so it was a natural dependency to leverage especially for python bindings. Unfortunately this is not always the case especially for older systems

[Roland Haas]

I see. An ET provided Boost ExternalLibrary would have to include all of Boost, both the STL template parts and the compiled parts, since we would not know what each thorn uses.

If the dependency persists the is should be listed in configuration.ccl along with GSL etc.

You may also need to consider an OPTIONAL dependency on CMake in particular if you require a non-ancient cmake. Some of CarpetX dependencies do that (AMReX, ADIOS I think) and can optionally use a cmake in ${CMAKE_DIR}/bin/cmake instead of just cmake. I think the two of them between them now require cmake 3.23 (from around July 2022, so very very recent for HPC clusters).

[Roland Haas]

I marked the thorns as DISABLED. We can include them like this for the next release an selectively enable them on clusters where we know Boos to be present (same as eg for PETSc based thorns). That is if the release managers are ok and the final vote in the ET call goes well.

[tootle]

@Roland Haas I had a look today and the dependencies that ptree relies within boost are too extensive to try and extract manually unfortunately. Therefore, the only option in the short term would be to include it as DISABLED if the ETK maintainers agree or to exclude these thorns altogether.

If the dependency persists the is should be listed in configuration.ccl along with GSL etc.

This would still require there to be an intermediate thorn that PROVIDES BOOST.

You may also need to consider an OPTIONAL dependency on CMake in particular if you require a non-ancient cmake. Some of CarpetX dependencies do that (AMReX, ADIOS I think) and can optionally use a cmake in ${CMAKE_DIR}/bin/cmake instead of just cmake. I think the two of them between them now require cmake 3.23 (from around July 2022, so very very recent for HPC clusters).

This is an interesting point. In reality the current version of KADATH is based on cmake 2.8 which I’ve recently had to start manually removing old safeties that interfere with methods implemented in cmake >3.2. Specifically, base KADATH included its on modules for find_package for its dependencies which would override the modern default scripts. This meant that all the search routines for looking for packages in $ENV{<PACKAGE NAME>_ROOT} were not searched since this was not implemented until cmake 3 I believe. I know there are those that unfortunately have to deal ith systems running old libraries, but this is not something I deal with at the moment so it is difficult to test and develop against.

[Wolfgang Kastaun]

@tootle Is the disabled kadath_pizza part simply using the EOS framework from WhiskyTHC? Because the RePrimAnd thorn that is part of the ET includes this functionality and the interface is very similar (I was the one who wrote the “pizza” parts of WhiskyTHC). So maybe it would make sense to just use RePrimAnd instead.

[Roland Haas]

I see.

Yes, an intermediate thorn (an ExternalLibrary) that provides Boost would be needed (right now, for CarpetX we use David Radice’s thorn: https://github.com/dradice/Boost, which provides a built in copy of Boost 1.55 or interfaces with a system installed Boost. It uses an older style of building things, than eg is used by “modern” ExternalLibraries like HDF5 or GSL, but actually that happens to work well for Boost).

Even if Boost is not part of the ET KadathThorn should declare a dependency on Boost in its configuration.ccl (same as the Reprimand helper thorn does), since this way a non-system Boost can be interfaced correctly (Cactus sets the relevant -I and -L options).

CMake is a mixed bag on clusters usually. It is very much geared towards workstation setups where there is exactly one copy of a library and its find_XXX functions seem to not always follow a set pattern or let a user override settings (autoconf is similar,but usually at least lets one override everything with CFLAGS and LDFLAGS which CMake resists). So I expect issues :-)

[tootle]

@Wolfgang Kastaun Thank you for your comment. Indeed, I’m aware of your work and have looked at the new RePrimAnd library a bit, however, I haven’t worked with the new thorn nor looked at what would be needed to interface the Fuka thorns with another EOS framework.

The kadath_pizza thorn is quite trivial, but plays a necessary role. For simulations that utilize a tabulated equation of state, the initial temperature and Ye need to be initialized prior to the ID_CONVERT stages. For our internal codes, the EOS manager initializes these values for us based on the extracted cold beta equilibrium slice, however, this was not the case when I was trying to run some comparison tests with WhiskyTHC. Instead of making the importer thorn more bloated I decided to write kadath_pizza to keep this functionality isolated and modular since interfaces differ between EOS frameworks.

[Roland Haas]

In the current KadathThorn, there is a .gitmodules file:

[submodule "src/fuka"]
    path = src/fuka
    url = https://bitbucket.org/fukaws/fuka/
[submodule "fuka"]
    branch = fuka

for the fuka library checked out into src/fuka as a git submodule. This all works and is fine.

However I notice that you are also including a section:

[submodule "fuka"]
    branch = fuka

which will be ignored since there is no fuka submodule (the submodule is src/fuka). Moreover, if it was not ignored, it would actually be an issue for the ET unless it is removed or fixed to a ET_YYYY_MM branch for each release branch and fixed to a specific git hash for each ET_YYYY_MM_vN release tag. Otherwise it points to the current head of that branch which is not what one would want for an ET release branch or tag (which should point to the specific, tested version that was released).

Since it is ignored anyway, it may be best to remove this section altogether to not give any false impression on what commit of fuka is actually checked out.

[tootle]

Hi @Roland Haas

Thanks for the feedback. Since I haven’t been told directly, what was the results of the meeting? Will the FUKA thorns be included as DISABLED? If so, is the following summary accurate for the required changes for inclusion:

1. Set the Kadaththorn to require BOOST
2. Remove submodule block
3. Generate ET release branches/tags (ET_2023_05) for fuka, kadaththorn, and kadathimporter repos
* For tags, what is used for vN?

Is this comprehensive? Thanks

‌

[Roland Haas]

1. yes, please
2. only the [submodule "fuka"] block (which is non-functional anyway)
3. you won’t need a ET_YYYY_MM etc. branch in fuka as long as the submodule import in KadathThorn is a “regular” one that specifies a specific commit (that one then needs to manually update when one wants to use a newer fuka version) and is not auto-set to track a (changing) branch (your’s does currently have a fixed commit). If things are changed and KadathThorn is set to track a specific branch of fuka (via the git submodule set-branch and git update --remote commands) then a ET_YYYY_MM branch and tags of type ET_YYYY_MM_vN (where N starts from 0 and increases by 1 for each point [bugfix] release) are needed. These are created shortly before the release when all code is frozen (I think about 1 week before the release). You may find it useful to grant one of the release managers commit rights (or at least make sure to approve pull requests by them). Keep in mind that there are about 83 git repos in the ET that they need to create branches in, so every little bit of help one can give them helps. See https://docs.einsteintoolkit.org/et-docs/Release_Process#Timeline_for_a_release_and_estimated_time_required and the “Release preparation II” item.

‌

[tootle]

I’ve resolved (1) and (2). For (3), thanks for the information related to using submodules in git.

Regarding the actions just prior to the release, I have no issue adding someone to the thorn repos to make their (our) lives easier if it allows for a level of automation. They can reach out to me and let me know (here or e-mail) Re: what they prefer and I’ll make it happen.

Thanks a lot Roland!

[Roland Haas]

@Samuel Cupp @Leonardo Werneck ?

[Samuel Cupp]

Sorry, but I didn’t really follow the specifics of point 3, as I haven’t dealt with anything that doesn’t just have a branch called “ET_release_date”. I am fine with having commit access so I can make the branches. If there’s something special that needs to be done for some repos, is there a guide or something on the wiki about these cases?

[Roland Haas]

The only docs that exist are: https://docs.einsteintoolkit.org/et-docs/Release_Process#create_branches

The script shown there should work for the Kadath codes in this ticket due to (3). They are basically the same as Kranc and its Examples repo (also a submodule with fixed git hash).

You will have to (at this point) manually handle proposal_ET_2023_05 in canuda/scalar since that is not the “default” branch and script fragment on the wiki assumes that one branches of release branches from the current head of the default branch. This most likely can (and should) be changed so that one can run the script fragment in the repository structure created by GetComponents on the master thornlist from the manifest repo and make it assume that the branching happens off the currently checked out branch (which for canuda/scalar is currently proposal_ET_2023_05). This (probably) just means removing the:

default_branch=$(git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@')

and

git checkout $default_branch

lines. It will also affect how one learns which changes to announce (which also looks at the default branch), but that process requires some human interaction anyway.

[tootle]

@Roland Haas should I add the branch/tags or is this something that needs to be managed by release managers? If the later, the ET branch should be made from the fukav2.1 branch for the FUKA source files as this is the branch all of the testing for ETK inclusion was done with.

[Samuel Cupp]

It’s fine if you make the branch, though we will be making branches for all the repos we have access to. If you don’t want it to be from master/main, it might be best if you handle it, but we can always do it as well. The branches aren’t scheduled to be made till 5-17, so there’s still a few weeks before we generate all the new release branches.

[tootle]

@Samuel Cupp I took care of the branches and tags this morning for all three repos.

[Samuel Cupp]

These thorns are now included as part of the Toolkit in the ET_2023_05 release.