Proposal: Add BaikalETK to the Toolkit

Create issue
Issue #2299 new
Zach Etienne created an issue

BaikalETK is an open-source NRPy+-based BSSN thorn, which has been demonstrated to yield excellent agreement with ML_BSSN when evolving BBHs and BNSs, and in certain cases (e.g., ADM mass volume integral over entire grid vs time in a pre-merged BNS simulation) better results than ML_BSSN.

BaikalETK is BSD 2-clause open-source licensed and available in the NRPy+ github page (https://github.com/zachetienne/nrpytutorial). One can browse its source code & documentation with nbviewer: https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/Tutorial-BaikalETK.ipynb

BaikalETK has the following neat features:

  1. The core BSSN kernel is generated by NRPy+/SymPy, which is entirely free and open source, just like BaikalETK. According to benchmarks by Erik, BaikalETK runs more than 10% faster than the Kranc/Mathematica -generated ML_BSSN currently part of the Toolkit. Work is ongoing to improve BaikalETK's performance even more.
  2. It is documented in a Jupyter notebook, which itself is linked to other Jupyter notebooks that document the BSSN formalism & gauge choices.
  3. It can be generated by either running the aforementioned Jupyter notebook or just importing a Python module within NRPy+. The Jupyter notebook contains a self-validation test that will fail if the Python module is updated without a corresponding update to the Jupyter notebook.
  4. Its scheduling and general structure are closely based on the very nice Lean code.
  5. It supports arbitrary finite-differencing order and a variety of gauge conditions, which can be extended further. It implements Brown’s covariant formulation of the BSSN equations.
  6. It supports as an option nonzero BSSN source terms, with appropriate linkage to TmunuBase.

Possible complications:

  1. Unlike ML_BSSN, which (due to Mathematica licensing constraints and consistency with other monolithic thorns) needed to contain all gauge choices, finite difference orders, etc., BaikalETK will only generate a BSSN thorn with a particular gauge and finite-differencing order chosen at the time of thorn generation. That is to say, choices about finite difference order, etc must be made prior to compile-time, which is not conventional at the moment.
  2. BaikalETK is not a complete ML_BSSN replacement yet as:

    1. BaikalETK does not yet support symmetries, but this should be rather straightforward to fix in an automated way, as NRPy+ requires information about gridfunctions and gridfunction symmetries.
    2. BaikalETK does not support Llama grids, and I’m not sure the path forward in this sense

Comments (5)

  1. Ian Hinder

    Hi Zach,

    I was interested to read about BaikalETK. It's great to see development in this area, and while of course I have quite an attachment to Kranc (and less so to ML_BSSN), it's good to see a modern automatically generated BSSN solver that doesn't rely on proprietary software!

    BaikalETK is an open-source NRPy+-based BSSN thorn, which has been demonstrated to yield excellent agreement with ML_BSSN when evolving BBHs and BNSs, and in certain cases (e.g., ADM mass volume integral over entire grid vs time in a pre-merged BNS simulation) better results than ML_BSSN. BaikalETK is BSD 2-clause open-source licensed and available in the NRPy+ github page (https://github.com/zachetienne/nrpytutorial). One can browse its source code & documentation with nbviewer: https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/blob/master/Tutorial-BaikalETK.ipynb BaikalETK has the following neat features: • The core BSSN kernel is generated by NRPy+/SymPy, which is entirely free and open source, just like BaikalETK. According to benchmarks by Erik, BaikalETK runs more than 10% faster than the Kranc/Mathematica -generated ML_BSSN currently part of the Toolkit. Work is ongoing to improve BaikalETK's performance even more.

    Do you know whether this is because the same equations are implemented more efficiently, or because the formulation is inherently more efficient? I'd be interested to hear more about Erik's benchmark results.

    Of course, in production BBH simulations, you usually see a depressingly small percentage of the runtime in the BSSN solver, but that's a topic for a different discussion!

    Possible complications: • Unlike ML_BSSN, which (due to Mathematica licensing constraints and consistency with other monolithic thorns) needed to contain all gauge choices, finite difference orders, etc., BaikalETK will only generate a BSSN thorn with a particular gauge and finite-differencing order chosen at the time of thorn generation. That is to say, choices about finite difference order, etc must be made prior to compile-time, which is not conventional at the moment.

    I didn't understand this. What sort of Mathematica licensing constraints would affect the requirement for ML_BSSN to contain all gauge choices? I thought that this was a performance-based decision by Erik in the early days. The idea was that you want to be able to use quantities computed already in the other RHSs also in the gauge condition, so the gauge condition needs to be in the same loop. Since you don't want to have multiple main evolution RHSs, one for each gauge condition, this means that all gauge conditions need to be in the main evolution loop. The Penn State BSSN code that I wrote, which was the evolution of the code I wrote for my PhD, had the gauge conditions as separate calculations, and could easily have split them out into other thorns if necessary. For finite difference orders, the original ML_BSSN used to have different thorns for the different orders. It was a relatively "recent" innovation to be able to generate multiple finite difference orders within the same thorn in Kranc, which is obviously a lot cleaner.

    • BaikalETK is not a complete ML_BSSN replacement yet as: • BaikalETK does not yet support symmetries, but this should be rather straightforward to fix in an automated way, as NRPy+ requires information about gridfunctions and gridfunction symmetries. • BaikalETK does not support Llama grids, and I’m not sure the path forward in this sense

    Essentially, for Llama, you need to support symmetries first, as Llama interpatch boundaries are implemented as symmetries. Then, you need to replace all derivatives with derivatives multiplied by the Jacobian matrix provided by the Coordinates thorn. You can have a look at the generated ML_BSSN code to see how this can be done.

    -- Ian Hinder Research Software Engineer University of Manchester, UK

  2. Erik Schnetter

    On Thu, Nov 7, 2019 at 8:16 PM Ian Hinder ian.hinder@manchester.ac.uk wrote:

    I compared three codes: (1) ML_BSSN as in the ET, (2) BaikalETK, and (3) a newer version of ML_BSSN that uses explicit loop tiling, inspired by our work on DG methods. I found that (2) was faster than (1) by about 10%-20%, and that (3) was faster than (2) by about 10%-20% in the one benchmark I ran (on Intel Skylake). I assume (but I did not check) that BaikalETK splits the loops differently than ML_BSSN, which is then faster either due to instruction or data cache effects.

    I have shared my findings with others (but apparently not on the ET mailing list – sorry!), and I now think that this loop tiling is the way to go. It is also significantly faster for hydro, and improves performance on Intel's KNL. I've sketched the implementation to Zach, and I hope that we can have an even faster BaikalETK in the future. I'm also looking forward to advertising some of the optimizations we built into Kranc to Zach – he has a similar optimization stage already, so we'll have to see whether important ones are missing.

  3. Log in to comment