Update: Add Piecewise Polytrope EoS Support to IllinoisGRMHD, Improved TOV Solver?

[Zach Etienne]

Piecewise-Polytrope Equation of State Support in IllinoisGRMHD

Leonardo (Leo) Werneck (a student currently working in my research group) recently documented much of IllinoisGRMHD in Jupyter notebooks and worked with me to add piecewise-polytrope equation of state support to IllinoisGRMHD. The code is open-source and available here:

https://github.com/zachetienne/nrpytutorial/tree/master/IllinoisGRMHD/doc

In addition, the Jupyter notebooks may be browsed via nbviewer:

https://nbviewer.jupyter.org/github/zachetienne/nrpytutorial/tree/master/IllinoisGRMHD/doc/

The updated code agrees to roundoff error with the original IllinoisGRMHD when a single polytrope is chosen, and the central density vs time when evolving piecewise-polytrope TOV stars (we tried the APR4 piecewise-polytrope as described in Read et al.) converges to zero density drift a bit above 2nd order with increased resolution, just like in the single-polytrope case. Further, we have successfully evolved SLy piecewise polytrope BNS initial data through a couple orbits (so far).

One can generate the new piecewise-polytrope enabled IllinoisGRMHD thorn family by cloning the above github repo and running the generate_illinoisgrmhd_from_ipynb_files.sh script in that directory. The IllinoisGRMHD/test directory contains some example parameter files.

Piecewise-Polytrope TOV Solver?

To perform the above piecewise-polytrope TOV tests, we needed a simple TOV solver with this capability (TOVSolver only supports single polytropes). To address this problem, Leo extended a standalone, numpy/Python-based piecewise-polytrope TOV solver that Phil Chang and I wrote, which can now build piecewise-polytropes (PP) from a large dictionary of PP EoSs. Most notably, the TOV system of ODEs remain in geometrized units, which makes them easier to read, and he wrote an EoS converter to rescale the CGS units to geometrized units where density = 1e15g/cm^3 = 1.0. The TOV solution file can be read into the Toolkit via the new NRPyPlusTOVID thorn that we wrote.

We would be happy to contribute this solver & thorn to the Toolkit as well – more recently Leo wrote & validated a C version of the TOV solver so we wouldn’t even need the TOV solution data file from the Python solver.

[Roland Haas]

Roland Haas is reviewer.

[Roland Haas]

In version 00fc953 "Merge branch 'master' of https://github.com/zachetienne/nrpytutorial" of nrpytutorial I get the following failures when trying to run generate_IllinoisGRMHD_from_ipynb_files.sh:

Traceback (most recent call last):
  File "/usr/bin/jupyter-nbconvert", line 11, in <module>
    load_entry_point('nbconvert==5.6.1', 'console_scripts', 'jupyter-nbconvert')()
  File "/usr/lib/python3/dist-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/usr/lib/python3/dist-packages/nbconvert/nbconvertapp.py", line 340, in start
    self.convert_notebooks()
  File "/usr/lib/python3/dist-packages/nbconvert/nbconvertapp.py", line 510, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/usr/lib/python3/dist-packages/nbconvert/nbconvertapp.py", line 481, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/usr/lib/python3/dist-packages/nbconvert/nbconvertapp.py", line 410, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/lib/python3/dist-packages/nbconvert/exporters/exporter.py", line 179, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/usr/lib/python3/dist-packages/nbconvert/exporters/exporter.py", line 197, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/usr/lib/python3/dist-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/lib/python3/dist-packages/nbconvert/exporters/exporter.py", line 139, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/lib/python3/dist-packages/nbconvert/exporters/exporter.py", line 316, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/lib/python3/dist-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/usr/lib/python3/dist-packages/nbconvert/preprocessors/execute.py", line 403, in preprocess
    with self.setup_preprocessor(nb, resources, km=km):
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/nbconvert/preprocessors/execute.py", line 345, in setup_preprocessor
    self.km, self.kc = self.start_new_kernel(**kwargs)
  File "/usr/lib/python3/dist-packages/nbconvert/preprocessors/execute.py", line 296, in start_new_kernel
    kc.wait_for_ready(timeout=self.startup_timeout)
  File "/usr/lib/python3/dist-packages/jupyter_client/blocking/client.py", line 120, in wait_for_ready
    raise RuntimeError('Kernel died before replying to kernel_info')
RuntimeError: Kernel died before replying to kernel_info

and

  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/dist-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/usr/lib/python2.7/dist-packages/traitlets/config/application.py", line 663, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-121>", line 2, in initialize
  File "/usr/lib/python2.7/dist-packages/traitlets/config/application.py", line 87, in catch_config_erro
r
    return method(app, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 469, in initialize
    self.init_sockets()
  File "/usr/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 260, in init_sockets
    self.init_iopub(context)
  File "/usr/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 265, in init_iopub
    self.iopub_port = self._bind_socket(self.iopub_socket, self.iopub_port)
  File "/usr/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 181, in _bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 547, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
    raise ZMQError(errno)
ZMQError: Address already in use

potentially indicating failure to handle a port already in use (in my case, port 8888 is never available since it is permanently claimed by a server process other than jupyter).

[Roland Haas]

At this point, in IllinoisGRMHD, while I have not fully dug down into the code it seems that the changes do no more harm that usual development for new features. Ie I can see that there may be roundoff / truncation level differences due to updated equations, there is nothing that obviously will break things.

Comments:

• instead of hand-crafting parallelism in generate_IllinoisGRMHD_from_ipynb_files.sh it may be better to provide a makefile. Note that the current script likely does not what you expect it does. Namely “wait” waits for all child processes to finish no any. “Proper” parallelism could be achieved by using (GNU) xargs' -P option, though make seems preferable to me.
• generate_IllinoisGRMHD_from_ipynb_files.sh modifies the input notebooks (in the sed line). It really should not do so.
• Currently removing the output thorn directories Convert_to_HydroBase and ID_converter_ILGRMHD and running generate_IllinoisGRMHD_from_ipynb_files.sh does not regenerate all files, ie. there are hand-written files mixed in with the auto-generated ones (license, README, documentation.tex). It would be better if this was not done.
• A duplicate of the notebook is included in the thorn’s doc directory and that duplicate is not auto-generated. This will lead to inconsistencies and should be avoided.
• generate_illinoisgrmhd_from_ipynb_files.sh only generated Convert_to_HydroBase and ID_converter_ILGRMHD but not the actual IllinoisGRMHD thorn. Update after call: I now see where they are generated.
• Some places significantly (and to me surprisingly) simplify the code eg:

static CCTK_REAL pressure_rho0_w(CCTK_REAL rho0, CCTK_REAL w)
{
  return((gamma_th /* <- Should be local polytropic Gamma factor */ -1.)*(w - rho0)/gamma_th /* <- Should be local polytropic Gamma factor */ ) ;
}

compared to

static CCTK_REAL pressure_rho0_w(eos_struct eos, CCTK_REAL rho0, CCTK_REAL w)
{

  // Set up Gamma_th:
#ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
  DECLARE_CCTK_PARAMETERS;
#endif

  // Compute P_cold, eps_cold
  CCTK_REAL P_cold, eps_cold;
  compute_P_cold__eps_cold(eos,rho0, P_cold,eps_cold);

  /* Compute the pressure as a function of rho_b (rho0) and
   * w = u + rho_b + p, using our hybrid EOS:
   *  ----------------------------------------------------------------------------
   * | p(rho_b,w) = ( P_cold + (Gamma_th-1)*( w - rho_b*(1+eps_cold) ) )/Gamma_th |
   *  ----------------------------------------------------------------------------
   */
  return( (P_cold + (Gamma_th-1.0)*( w - rho0*(1.0+eps_cold) ) )/Gamma_th );
}

• there’s a couple of “// FIXME: only for single gamma-law EOS.” comments. Are those to be worried about?

‌

[Roland Haas]

Right now this has not yet been merged into master.

To make some progress, I created a branch ET_2020_04_candidate of WVUThorns in https://bitbucket.org/rhaas80/wvuthorns .

The code compiles however it fails in the testsuite with:

WARNING[L1,P0] (Seed_Magnetic_Fields): Boolean variable is set with integer: Seed_Magnetic_Fields::enable_IllinoisGRMHD_staggered_A_fields = 1

WARNING[L1,P1] (Seed_Magnetic_Fields): Boolean variable is set with integer: Seed_Magnetic_Fields::enable_IllinoisGRMHD_staggered_A_fields = 1

WARNING[L1,P0] (Cactus): Major error in parameter file '/data/rhaas/postdoc/gr/cactus/ET_Next/arrangements/WVUThorns/IllinoisGRMHD/test/magnetizedTOV.par' line 216: Parameter 'ID_converter_ILGRMHD::Gamma_Initial' not found
WARNING[L1,P0] (Cactus): Major error in parameter file '/data/rhaas/postdoc/gr/cactus/ET_Next/arrangements/WVUThorns/IllinoisGRMHD/test/magnetizedTOV.par' line 217: Parameter 'ID_converter_ILGRMHD::K_Initial' not found
WARNING[L1,P1] (Cactus): Major error in parameter file '/data/rhaas/postdoc/gr/cactus/ET_Next/arrangements/WVUThorns/IllinoisGRMHD/test/magnetizedTOV.par' line 216: Parameter 'ID_converter_ILGRMHD::Gamma_Initial' not found
WARNING[L1,P1] (Cactus): Major error in parameter file '/data/rhaas/postdoc/gr/cactus/ET_Next/arrangements/WVUThorns/IllinoisGRMHD/test/magnetizedTOV.par' line 217: Parameter 'ID_converter_ILGRMHD::K_Initial' not found
WARNING level 0 from host ekohaes8 process 0
  in thorn Cactus, file /data/rhaas/postdoc/gr/cactus/ET_Next/configs/sim/build/Cactus/main/ProcessParameterDatabase.c:195:
  -> CCTKi_SetParameterSetMask: 2 major errors in parameter file
WARNING level 0 from host ekohaes8 process 0
  in thorn Cactus, file /data/rhaas/postdoc/gr/cactus/ET_Next/configs/sim/build/Cactus/main/ProcessParameterDatabase.c:195:
  -> CCTKi_SetParameterSetMask: 2 major errors in parameter file
WARNING level 0 from host ekohaes8 process 1
  in thorn Cactus, file /data/rhaas/postdoc/gr/cactus/ET_Next/configs/sim/build/Cactus/main/ProcessParameterDatabase.c:195:
  -> CCTKi_SetParameterSetMask: 2 major errors in parameter file

Those parameters were removed in the commit. Since removing parameters is a regression it is required to be announced one release before the actual removal, meaning that the commit as proposed cannot be included in the ET.

[Zach Etienne]

The parameters aren’t removed – rather they were moved as part of the EOS clean up/generalization. Originally they were duplicated between IllinoisGRMHD and ID_converter_ILGRMHD, so you had to set them twice. Now you only have to set them once. So I’d also classify this partly as a bugfix.

[Roland Haas]

That still counts as “removed”. Namely a parameter file (the test) that worked before, now fails.

[Zach Etienne]

What is the accepted way to deal with this?

It’s trivial to update the parameter file to make it work again. One idea: put the parameters back in so that a more detailed error message may be provided to the user, explaining how to update their parameter file?

Longer term I think the whole ID_converter_ILGRMHD thorn should be deprecated – its functionality merged into IllinoisGRMHD proper.

[Roland Haas]

Unfortunately even requiring a trivial modification is not acceptable. This is partially due to what is trivial tending to depend to be on the level of experience a user has with the code in question, ranging from the author, for whom everything is trivial, to someone trying to run a gallery parfile or example parfile from the web, for whom nothing is trivial.

An acceptable way would be the following:

• Re-add the parameters
• In this release introduce an additional parameter “eos_params_from_illgrmhd” to ID_converter_ILGRMHD which must default to “false”. Have ID_converter_ILGRMHD use either its own parameter value or ILGRMHD’s values depending on the value of this new parameter
• in the 2020_04 release mark this new parameter and the two that were removed as “deprecated”
• in the 2020_10 release remove all three deprecated parameters as well as the extra code

To retire ID_converter_ILGRMHD it also needs to be deprecated for one release. Note that we typically retire code only once it no longer functions, not when it is not longer considered interesting, both to allow very old parfiles to run and also since we never quite know what the ET community is still using.

As far as I can tell there is no way to put the parameters back in and warn a user about them not be supported anymore (b/c one cannot change the default values and must accept runs with the default values for only one of the pair of corresponding parameters).

Policy on retiring functionality is outlined on the wiki: https://docs.einsteintoolkit.org/et-docs/Policies_to_retire_functionality

Thorns and parameters probably should not stay deprecated for very long before being removed (ie don’t just mark everything as deprecated to get a blank cheque on removing anything anytime) but given https://docs.einsteintoolkit.org/et-docs/Deprecated_features that has definitely happened in the past.

[Roland Haas]

Is this being superseded by #2690 ? If so it should be closed as a duplicate.

[Zach Etienne]

This functionality will be added in an upcoming GRHayLET thorn. IllinoisGRMHD will remain untouched.