Add GiRaFFE to the Einstein Toolkit

Create issue
Issue #2064 resolved
Zach Etienne created an issue

We (the authors of GiRaFFE) hereby request that GiRaFFE be added to the Einstein Toolkit. In short, GiRaFFE is the first open source, dynamical-spacetime code for modeling plasma flows in general relativistic force-free electrodynamics (GRFFE). It will be the best code in the Toolkit for modeling black hole and neutron star magnetospheres, as these magnetically-dominated regions are best modeled numerically by a GRFFE code (GRMHD codes break down when modeling very highly magnetized plasmas).

Here is the commit ID for code reviewer comments:

Here is the original release announcement to the mailing list, including link to the code release paper:

We are pleased to announce the public release of GiRaFFE, an open-source General Relativistic Force-Free Electrodynamics (GRFFE) code for dynamical spacetimes.

-= Code Release Paper =-

Our code release paper--including a full description of the GiRaFFE code, a discussion of potential applications of GiRaFFE, and code validation test results--has been uploaded to the arXiv: .

-= Executive Summary =-

GiRaFFE adopts the strategy pioneered by McKinney [1] and modified by Paschalidis and Shapiro [2] to convert a general relativistic magnetohydrodynamic (GRMHD) code into a GRFFE code. In short, GiRaFFE exists as an extensive modification to IllinoisGRMHD [3]. Both GiRaFFE and IllinoisGRMHD leverage the Einstein Toolkit’s highly-scalable infrastructure [4,5] to make possible large-scale simulations of magnetized plasmas in strong, dynamical spacetimes on adaptive-mesh refinement (AMR) grids.

As you are aware, IllinoisGRMHD is an open-source rewrite of the Illinois Numerical Relativity Group's GRMHD code. GiRaFFE, on the other hand, is not based on any existing GRFFE code and is designed only to solve the equations of GRFFE in the context of fixed or dynamically-generated spacetimes. GiRaFFE has passed a large suite of GRFFE code validation tests passed by other state-of-the-art GRFFE codes, leading us to conclude that GiRaFFE is now ready for large-scale, state-of-the-art GRFFE simulations in dynamical spacetimes.

-= New Thorns for the Einstein Toolkit =-

All GiRaFFE thorns are listed below. Next to the name of each thorn, we include the number of lines of code to highlight the fact that GiRaFFE is quite compact and designed to be welcoming to new users. In addition, the comment-to-line-of-code ratio is very nearly 50%, and the code release paper provides a clean, broad-brush overview of the formalism and algorithms adopted. To become acquainted with any given thorn, first take a look at its README file.

  • GiRaFFE (~1,600 lines of code): Solves the GRFFE equations in the formalism of Paschalidis & Shapiro [2] using the same reconstruction (PPM) and staggered vector potential methods as IllinoisGRMHD.
  • GiRaFFEfood (~1,300 lines of code): Code validation initial data and basic diagnostics. Contains Five 1D tests in flat spacetime (fast wave, Alfvén wave, degenerate Alfvén wave, "three-wave", FFE breakdown), One 3D test in flat spacetime (aligned rotator), and Three 3D tests in curved spacetimes ("Exact Wald" solution, "Magnetospheric Wald" solution, and split monopole).
  • ShiftedKerrSchild (~230 lines of code): Sets up all 3+1 quantities for a Kerr-Schild spacetime with an arbitrary radial shift, to mitigate effects of extreme spacetime curvature near r_{KS}=0. See Appendix of code release paper for full discussion.
  • VolumeIntegrals_slab_Gfood (~310 lines of code): Performs volume integrals over arbitrary rectangular prisms as specified in parameter files.
  • GiRaFFE_to_HydroBase & ID_converter_GiRaFFE (~230 lines of code): Acts as a translation layer to convert 3-velocities in HydroBase's Valencia formalism to/from the 3-velocity found in the induction equation (v^i = u^i/u^0; native to GiRaFFE).

-= License =-

To maximize compatibility with existing thorns in ET, all of the aforementioned GiRaFFE thorns are either licensed GPLv2+ or 2-clause BSD (FreeBSD). See LICENSE file in each.

-= Download GiRaFFE! =-

The collection of thorns comprising GiRaFFE may be downloaded here:


The GiRaFFE team

-Zach Etienne -Mew-Bing Wan -Maria Babiuc -Sean McWilliams -Ashok Choudhary

References: [1] J.C. McKinney. Mon. Not. R. Astron. Soc., 367:1797, 2006. [2] V. Paschalidis and S. L. Shapiro. Phys. Rev. D, 88:104031, 2013. [3] Z. B. Etienne, V. Paschalidis, R. Haas, P. M¨osta, and S. L. Shapiro. Class. Quantum Grav., 32(17):175009, 2015. [4] [5]

Keyword: GiRaFFE,
Keyword: GRFFE,
Keyword: IllinoisGRMHD

Comments (24)

  1. Ian Hinder

    Unfortunately, the code is not yet reviewed, so we have to bump it to the next release. We will try very hard to get this reviewed shortly after the release, added to master, and then included in the next release.

  2. Roland Haas
    • removed comment

    This is taking me a very long time to review, sorry. I think I will try and see if I can review just the differences to IllinosGRMHD (which has already been reviewed).

  3. Roland Haas
    • removed comment

    I added a diff of IllinoisGRMHD to GiRaFFE comparing either files of the same name (small changes only indeed) or names that are similar but one contains GRFFE in it (larger changes), and excluding the TeX documentation and README files. That brings the diff down to ~3200 lines.

  4. Roland Haas
    • changed status to open
    • removed comment

    Done with review of GiRaFFE proper, no "Please apply" since there are too many (small) issues that should at least be looked at. Nothing that I can point a finger at as being definitely wrong, though in some places comments and code contradict each other.

  5. Roland Haas
    • removed comment

    This was discussed in today's ET call and Roland has no objections about including GiRaFFE, GiRaFFE_food, ID_converter_GiRaFFE

  6. Zach Etienne reporter
    • removed comment

    GiRaFFE_to_HydroBase and ShiftedKerrSchild are also required thorns. The former populates HydroBase variables based on GiRaFFE variables -- needed for certain diagnostics. The latter is needed by GiRaFFEfood.

  7. anonymous
    • removed comment

    No objections against those two either (though I think some may be missing explicit documentation in their .tex files).

  8. Roland Haas
    • removed comment

    The thorns were included in git hash 62866e1 "GiRaFFE review:" of wvuthorns and added to the thornlist in b963885 "Add GiRaFFE thorn family" of the maninfest.

    Documentation tex files are still missing as far as I can tell.

  9. Roland Haas
    • removed comment

    Zach says (offline due to not yet having an external CCT account):

    Documentation for GiRaFFE is located in the GiRaFFE/doc directory (you'll see documentation.tex and references.bib). It can be built with the usual:

    pdflatex documentation.tex pdflatex documentation.tex bibtex documentation.aux pdflatex documentation.tex

    The documentation build system currently does not support BibTeX, which might explain why the GiRaFFE documentation isn't appearing on your radar.

  10. Roland Haas
    • removed comment

    Are you ok if I copy and paste you email with "Zach says" to the ticket?

    The GiRaFFE docs are fine (or at least there is nothing in there that I did not already comment on during the actual review for GiRaFFE). The docs I was referring to in the ticket at those for the helper thorns:

    • GiRaFFE_to_HydroBase
    • ShiftedKerrSchild

    Please note that I have not checked right now if there is actual docs for those two. In both cases the docs should be very short, for the ShiftedKerrSchild one should be able to just copy&paste from the GiRaFFE paper and for GiRaFFE_to_HydroBase this is really something that one may not even need since most likley you already have the info in the actual GiRaFFE docs (the separate thorn is only there so that one can run without HydroBase analysis and get a bit more speed?).

    These are very minor and should only be done after the WVUThorns issues you wanted to work on this week.

  11. Roland Haas
    • removed comment

    Zach says:

    I was tempted to mention this during today's telecon, but GiRaFFE_to_HydroBase is destined for extinction. One of the coauthors of the GiRaFFE paper is working to incorporate this thorn into GiRaFFE itself, so that GiRaFFE uses the HydroBase variables whenever possible, and when it is not possible, to update the HydroBase variables continuously (with very little additional cost, it turns out). It's still a WIP, but here's the pull request:

    Documentation for ShiftedKerrSchild should be a matter of simply copy-pasting some of the text from our paper (basic equations) into the doc/ directory. I'll put this on my to-do list after the outstanding WVUThorns_diagnostics items.

  12. Roland Haas

    What is the status of these? Is there anything already in need for review? Should the milestone be kept or removed?

  13. Roland Haas

    @Zach Etienne should we keep this open or close it until the pull request is ready for review? Having a milestone attached to it makes us revisit it each release.

  14. Zach Etienne reporter

    @Roland Haas We have decided to go in a different direction from this pull request. We are currently NRPy-fying GiRaFFE in a way that pulls each component of GiRaFFE into a unit test that validates against the original version. Having the entire GiRaFFE written & documented in Jupyter notebooks (where one need only run the Jupyter notebooks to generate the entire thorn/thorn family, and every equation is NRPy-fied + documented with a hyperlink to the relevant paper) will be most sustainable moving forward.

  15. Ian Hinder

    @Zach Etienne Have you found it easy to use version control with Jupyter notebooks? Is it a good idea to pull the whole code into notebooks? I don’t know how sustainable it is when you have multiple contributors; merging changes from different people can become very difficult.

  16. Zach Etienne reporter

    @Ian Hinder : Version control with Jupyter is only slightly more difficult than with plain source code, as its JSON is plaintext.

    Is it a good idea to pull the whole code into notebooks? I don’t know how sustainable it is when you have multiple contributors; merging changes from different people can become very difficult.

    Writing the Jupyter notebook is only the first step in NRPy+ development, but arguably the most important as it contains the documentation and some validation checks. I’d liken the Jupyter notebook phase more to collaboratively writing a journal article with a shared git repo than code development within such a repo.

    After the notebook has been written, a separate Python module containing the code developed in the Jupyter notebook is then written. At the bottom of every NRPy+ Jupyter notebook it is required to include a self-validation check against the separate Python module (many NRPy+ Jupyter notebooks have additional validation checks). Further, all NRPy+ Python modules must include proper unit testing on all symbolic expressions generated, so that Travis CI can confirm agreement with the trusted version. This has the nice consequence of often requiring developers to update the documentation any time the module is updated in order for the Jupyter notebook to pass self-validation tests, though generally NRPy+ developers prefer updating the Jupyter notebooks as a first step, then propagating changes to the module & unit tests. Proper documentation makes our lives much easier and is the foundation of a sustainable codebase.

    The above approach has worked fine with multiple contributors so far, as NRPy+ is extremely modular, with 70+ Python modules & about 100 Jupyter notebooks. Also I generally don’t put multiple students on exactly the same project (though they certainly do interact and benefit from each others' documentation). Thus each user works on their own unique notebooks without worry of interfering with others, but knowing that their work in making the documentation excellent will help others. With this development approach, only after completing the notebook stage and well into the Python module stage is there any real worry of merge conflicts, but I’ve never had the problem of two developers trying to push on the same module at the same time.

  17. Log in to comment