Inclusion of GRHayL library and associated MHD thorns

[Samuel Cupp]

The General Relativistic Hydrodynamics Library (GRHayL) provides core functionality for IllinoisGRMHD-like codes in an infrastructure-agnostic form. The GRHayLib thorn sets up the core library in the Cactus Framework. Included in this update is the change to IllinoisGRMHD to use this new library, which also includes many improvements, simplifications, and other enhancements. Additionally, we provide the GRHayLHD thorn for purely hydrodynamic simulations which has noticeable speed improvements over the MHD code. For CarpetX, we provide IllinoisGRMHDX and GRHayLHDX.

To help with testing and future work, we also provide GRHayLID and GRHayLIDX, which set up simple MHD data for HydroBase (e.g. the Balsara tests). Note that since GRHayLib only provides the library, one can use GRHayLib/GRHayLID to set up initial data and then use an EOS_Omni- or EOSX-based evolution thorn with no issues.

Part of the simplifications to IllinoisGRMHD make the ID_converter_ILGRMHD and Convert_to_HydroBase thorns obsolete, and they are deprecated in this release. Special deprecation code is scheduled to ensure the core behavior is maintained till the next release cycle which checks the parameters of these auxiliary thorns.

[Samuel Cupp]

@Peter Diener is the reviewer

[Samuel Cupp]

To check out the Carpet-based thorns, simply add

!TARGET   = $ARR
!TYPE     = git
!URL      = https://github.com/GRHayL/GRHayL.git
!REPO_BRANCH = main
!REPO_PATH= implementations/$2
!CHECKOUT =
GRHayL/GRHayLib

!TARGET   = $ARR
!TYPE     = git
!URL      = https://github.com/GRHayL/GRHayLET.git
!REPO_BRANCH = main
!REPO_PATH= $2
!CHECKOUT =
GRHayLET/GRHayLHD
GRHayLET/GRHayLMHD
#GRHayLET/IllinoisGRMHD
GRHayLET/GRHayLID

Note that GRHayLMHD is the current name of IllinoisGRMHD on the main branch, as I am still comparing with IllinoisGRMHD and wanted to compile them together. In the release, it will be IllinoisGRMHD, and the thorn in the WVUThorns arrangement will be removed.

[Samuel Cupp]

The CarpetX thorns are also in the GRHayLET repo, but require CarpetX to compile. They are

GRHayLET/GRHayLIDX
GRHayLET/GRHayLHDX

GRHayLET/IllinoisGRMHDX is still in a separate branch being debugged, and I will update when it is merged or if it won’t make it in time for the release.

[Peter Diener]

I added the repository at the end of my einstein toolkit thornlist and got errors from GetComponents:

Do you want to update all existing components? yes, no [no] : n
-----------------------------------------------------------------
  Checking out module: Scalar/doc
      from repository: https://bitbucket.org/canuda/Scalar.git
                 into: Cactus/arrangements
-----------------------------------------------------------------
  Checking out module: GRHayL/GRHayLib
      from repository: https://github.com/GRHayL/GRHayL.git
                 into: Cactus/arrangements
-----------------------------------------------------------------
  Checking out module: GRHayLET/GRHayLHD
      from repository: https://github.com/GRHayL/GRHayLET.git
                 into: Cactus/arrangements

Warning: Could not checkout GRHayLET/GRHayLHD, unable to switch to branch main. Any existing symlinks to GRHayLET/GRHayLHD will be broken

-----------------------------------------------------------------
  Checking out module: GRHayLET/GRHayLMHD
      from repository: https://github.com/GRHayL/GRHayLET.git
                 into: Cactus/arrangements
-----------------------------------------------------------------
  Checking out module: GRHayLET/GRHayLID
      from repository: https://github.com/GRHayL/GRHayLET.git
                 into: Cactus/arrangements

Warning: Could not checkout GRHayL/GRHayLib, unable to switch to branch main. Any existing symlinks to GRHayL/GRHayLib will be broken

-----------------------------------------------------------------
  1 components checked out.
  0 components updated.

  Unable to process GRHayLET/GRHayLHD
  Unable to process GRHayL/GRHayLib

Summary of Warnings:

Could not checkout GRHayLET/GRHayLHD, unable to switch to branch main. Any existing symlinks to GRHayLET/GRHayLHD will be broken
Could not checkout GRHayL/GRHayLib, unable to switch to branch main. Any existing symlinks to GRHayL/GRHayLib will be broken

‌

[Peter Diener]

The code compiled okay, but I have trouble running the Balsara1 tests in GRHayLHD and GRHayLMHD. First of all both tests tried to set a parameter from GRHayLID called test_1D_initial_data. However, the parameter defined there is initial_data_1D. In addition they tried to set test_shock_direction, whereas the parameter defined in GRHayLID is shock_direction.After fixing those, the tests started running but failed when trying to do output of GRHayLHD::rho_b and GRHayLHD::pressure (similarly for GRHayLMHD). Looking in the interface.ccl of the hydro thorns is see rho_b and pressure mentioned in a comment but they are not defined.

‌

[Samuel Cupp]

Sorry, I was actually working on that right now. Also, there’s been some internal pressure to make sure this gets in this release, but I am not sure how feasible it is to actually get them all ready. However, I at the very least would like to see if I can get GRHayLib, GRHayLID, and GRHayLHD in.

I did push changes that should make the Balsara1 test work. We had switched the thorn from using its own copies of HydroBase to directly using HydroBase variables, which led to the issues you saw in regards to rho and P.

[Samuel Cupp]

I ported all the changes made to GRHayLHD and GRHayLID to the X versions and added a test to GRHayLHDX. I also checked that the test looks the same for both versions (as much as they can with cell/vertex differences).

[Peter Diener]

The tests for GRHayLHD now runs, but the new data is for 1 processor whereas the test requirement is for running on 2 processors. The test for GRHayLMHD still sets wrong parameters. After fixing those, the test fails with differences in all output files.

GRHayLib has no documentation. I would think that a library in order to be really useful would have documentation that describe the provided interfaces.

‌

‌

[Samuel Cupp]

Sorry if I was unclear before, but we aren’t trying to get the MHD into this release anymore. Just GRHayLib, GRHayLHD(X), and GRHayLID(X). I have documentation.tex that I have been working on, but I haven’t finished and pushed it yet. I will update as soon as I do.

[Samuel Cupp]

I pushed a version of the documentation for GRHayLib, as well as documentation updates/additions for the others.

[Samuel Cupp]

For the full library documentation, I point them to the GitHub wiki where we give details on all the library functions. Hopefully that is sufficient.

[Peter Diener]

I have compiled with both the gnu and intel compilers on my laptop and find that the test “Balsara1” fails with the intel compilers but passes with the gnu compilers. Looking at this in more details, it looks to be an issue with the initial data setup. In the parameter file, the discontinuity position is set to zero. With the gnu compilers this results in the pressure being 1 at x=0 and 0.1 at x=0.000625. With the intel compilers the pressure is 1 at x=-0.000625 and 0.1 at x=0. So somehow the iniitlal profile is shifted by dx. Looks like there is some sensitivity to difference in rounding between the compilers.

[Roland Haas]

Fails with GNU for balsara0 still on testing server https://einsteintoolkit.github.io/tests/build_738.html https://github.com/EinsteinToolkit/tests/blob/gh-pages/records/version_738/sim_738_2/GRHayLHD/Balsara0.diffs

[Samuel Cupp]

The Balsara0 test is passing now in the CI. Let me know if there’s any issues.

Since CarpetX is in now, I enabled GRHayLIDX and GRHayLHDX so the tests can run.

[Samuel Cupp]

GRHayLib, GRHayLHD(X), and GRHayLID(X) are part of the Einstein Toolkit as of the ET_2023_11 release.