1. The Enzo Project
  2. Untitled project
  3. enzo-dev
  4. Pull requests

Pull requests

#194 Merged at 1203424
Repository
enzo-grackle
Branch
week-of-code
Repository
enzo-dev
Branch
week-of-code

Grackle chemistry and cooling in Enzo

Author
  1. Britton Smith
Reviewers
Description

This PR adds support for the Grackle chemistry and cooling library to Enzo. Use of Grackle requires compiling with "make grackle-yes". All Grackle functionality should be inside #ifdef USE_GRACKLE, but please leave a comment if you find anything that is not.

As I mentioned in my email to enzo-dev a while ago, Grackle is a chemistry and cooling library that was originally derived from Enzo's MultiSpecies and Cloudy cooling modules. However, there have been a number of updates to the UV background and metal cooling functionality make this not only unique from the original implementation, but potentially a more attractive option for simulations without radiative transfer, such as:

  • UV backgrounds are treated via interpolating from data tables loaded from disk rather than piece-wise polynomial functions, making it somewhat easier to add support for new background models.

  • UV background and cooling data are contained within the same input file and are more consistent than the currently available Cloudy cooling data and Enzo UV background models. Currently, the Grackle distribution comes with UV background and cooling data for two different models:

    1. Faucher-Giguere et al. (2009).

    2. Haardt & Madau (2012).

  • Unlike the original Cloudy cooling which required separate input files for cooling before the UV background turns on and after, all data is contained in a single table. This means one no longer has to run with one input file to the redshift where the UV background starts, stop the simulation, and restart with another input file.

  • Also unlike the original Cloudy cooling module, Grackle supports the option to also solve the primordial cooling via interpolation from a table. Thus, one is no longer required to run with the MultiSpecies functionality in order to calculate the primordial component. This simplified method is somewhat faster and requires fewer baryon fields to be stored, lowering the ram and disk footprint.

This also enables the use of Enzo for the AGORA project with the required common physics package, a.k.a. Grackle.

What has been added:

  • support to solve chemistry cooling and calculate cooling time using Grackle functions
  • narrative documentation discussing Grackle in general, how to build Enzo with it, and a full description of parameters
  • sample cosmology simulation parameter file using Grackle in run/CosmologySimulation/AMRCosmology_Grackle

Tests

I ran a very small cosmological simulation using Cloudy cooling, Grackle with MultiSpecies 1, and Grackle with MultiSpecies 0 (fully-tabulated). Note, the input data are not the same for the Cloudy run and the Grackle runs. The Grackle data corresponds to a newer version of the Haardt & Madau background which starts at a higher redshift (z = 14.849 instead of z = 8.9). Below are projections of metallicity and phase plots of cell mass in bins of density and temperature for each of the three simulations. The strange centering in the projections is due to the fact that this is a pretty low-resolution nested simulation and we are viewing only the refinement region.

Cloudy Cooling

Cloudy Metallicity Cloudy phase plot

Grackle w/ MultiSpecies = 1

Grackle MS1 Metallicity Grackle MS1 phase plot

Grackle Fully-tabulated (note: phase plot y range is different)

Grackle Tabulated Metallicity Grackle Tabulated phase plot

The most notable different is the addition of a small amount of cold gas at roughly 1e3 K in the Grackle runs. I'm not entirely sure where exactly this comes from, but I don't think it is indicative of something incorrect.

Comments (23)

    1. Britton Smith author

      You can turn grackle on with the use_grackle parameter. However, I wanted to make a compile time option as well so as to not force people to install grackle if they never intended to use it.

      1. Nathan Goldbaum

        Makes sense. For some reason I was thinking you could do this outside the preprocessor but clearly everyone would need a grackle.h file for that to work.

        In the future we may want to come back to this and think about making grackle a required dependency, particularly if grackle starts to become more popular than enzo's built-in cooling routines.

  1. Sam Skillman

    I've tested this out, and once I actually follow the directions it works great! Thanks for all of this work, Britton Smith . Would you be up for a G+ dev hangout in the near future to have folks give feedback on the structure of enzo <--> grackle?

  2. chummels

    Hi Britton Smith,

    First, let me say that I'm very excited that you're PR'ing this, as I'm eager to use it for my own sims. Thank you for your hard work!

    I followed your installation instruction and was able to get grackle installed on TACC stampede without any problems. I then followed your instructions for installing and building grackle support in enzo. This was pretty straightforward too. Finally, I tried to run your AMRCosmology Test simulation in the run directory, but I'm getting failures. Essentially, it builds the ICs fine, runs ring ok, and then starts up the simulation, but it keeps repeating this line in STDERR:

    "Mean molecular weight not converged! NaN NaN"

    I found this in the Grackle Source for cool1d_multi_g.F, in that it cannot seem to converge on Mu when adding in the different metal species. But I'm not exactly sure why this is happening. I'm just running with all of your defaults. It looks like it gets past all of the initializations and is actually starting calculation to cause this error. Any ideas?

    Sam Skillman : were you able to run these test runs OK?

    1. chummels

      To follow up on this comment, I was able to get enzo-grackle installed and working on a Darwin box (mountain lion). Everything seemed to work fine after I made the environment variable modification noted below so that enzo could find the grackle libraries. But I still don't know what's going awry with grackle-enzo on Stampede as discussed above.

        1. Sam Skillman

          Sadly, I did have this error and managed to fix it (user error on my end), but I don't remember precisely what it was. I believe the key parts for me were:

          1. Don't just copy the Make.mach.whatever from my .enzo/ to .grackle/ . I think this is mostly because of the -fPIC and it didn't have the necessary install directory flags.
          2. Make sure to set the MACH_INSTALL_PREFIX to somewhere such that $(MACH_INSTALL_PREFIX)/lib is already in your LD_LIBRARY_PATH (DYLD for silly macs) or that you later add to your (DY)LD_LIBRARY_PATH.
          3. Make sure you actually do "make install" from grackle/src and not just "make"
          1. Britton Smith author

            chummels has correctly pointed out to me that the grackle documentation states that you can use your Enzo make file to build grackle, which is not true and seems to be the cause of these problems. I will correct this as soon as I can.

      1. Nathan Goldbaum

        I'm seeing the same issues as Cameron on Stampede with an independently generated makefile for grackle. No idea what's going wrong. Let me know if it would help to see my makefiles.

        1. chummels

          Yeah, I went back and generated a brand new Makefile on Stampede by modifying Make.mach.unknown. It too fails. The easiest way to demonstrate this is to (from grackle src/clib dir):

          $ make
          $ make install
          $ cd ../example
          $ make example
          $ ./example.exe
          Initializing chemistry data.
          Initializing Cloudy cooling: Metals.
          cloudy_table_file: ../../input/CloudyData_UVB=HM2012.h5.
          Cloudy cooling grid rank: 3.
          Cloudy cooling grid dimensions: 29 26 161.
          Parameter1: -10 to 4 (29 steps).
          Parameter2: 0 to 14.849 (26 steps).
          Temperature: 1 to 9 (161 steps).
          Reading Cloudy Cooling dataset.
          Reading Cloudy Heating dataset.
          Initializing UV background.
          Reading UV background data from ../../input/CloudyData_UVB=HM2012.h5.
          UV background information:
            Haardt & Madau (2012, ApJ, 746, 125) [Galaxies & Quasars]
            z_min =  0.000
            z_max = 15.130
          Setting UVbackground_redshift_on to 15.130000.
          Setting UVbackground_redshift_fullon to 15.130000.
          Setting UVbackground_redshift_off to 0.000000.
          Setting UVbackground_redshift_drop to 0.000000.
           NaN in edot[1]:            1           1           1                     NaN
            9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999996E-26
            9.9999997E-21  0.0000000E+00  0.0000000E+00  0.0000000E+00
           NaN in edot[1]:            2           1           1                     NaN
             1.815000      6.0916697E-03   1.615000      6.0916697E-03  6.0916697E-03
            6.0916697E-03   1.875000      2.5257232E+07  -84381.51
           NaN in edot[1]:            3           1           1                     NaN
            9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999996E-26
            9.9999997E-21  0.0000000E+00  0.0000000E+00  0.0000000E+00
           NaN in edot[1]:            4           1           1                     NaN
             1.815000      6.0916697E-03   1.615000      6.0916697E-03  6.0916697E-03
            6.0916697E-03   1.875000      2.5257232E+07  -84381.51
           NaN in edot[1]:            5           1           1                     NaN
            9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999996E-26
            9.9999997E-21  0.0000000E+00  0.0000000E+00  0.0000000E+00
           NaN in edot[1]:            6           1           1                     NaN
             1.815000      6.0916697E-03   1.615000      6.0916697E-03  6.0916697E-03
            6.0916697E-03   1.875000      2.5257232E+07  -84381.51
           NaN in edot[1]:            7           1           1                     NaN
            9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999996E-26
            9.9999997E-21  0.0000000E+00  0.0000000E+00  0.0000000E+00
           NaN in edot[1]:            8           1           1                     NaN
             1.815000      6.0916697E-03   1.615000      6.0916697E-03  6.0916697E-03
            6.0916697E-03   1.875000      2.5257232E+07  -84381.51
           NaN in edot[1]:            9           1           1                     NaN
            9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999997E-21  9.9999996E-26
            9.9999997E-21  0.0000000E+00  0.0000000E+00  0.0000000E+00
           NaN in edot[1]:           10           1           1                     NaN
             1.815000      6.0916697E-03   1.615000      6.0916697E-03  6.0916697E-03
            6.0916697E-03   1.875000      2.5257232E+07  -84381.51
          Segmentation fault (core dumped)
          

          Using gdb with this, it appears the error occurs when c++ passes pointers to fortran programs, specifically in the call to solve_rate_cool_g.F, many of the values it gets passed are recognized, which yield a quick segfault. I'm not sure how to fix this--perhaps some additional compiler flag in the Makefile? I'm sort of out of ideas here.

          The Makefile that I'm using is here: http://paste.yt-project.org/show/4237/

          1. Matt Turk

            My guess is that it's related to -fdefault-real-8 and -fdefault-double-8 in your makefile. Try seeing if it fails in 32 bit mode. Also, did you mean "many of the values it gets passed aren't recognized"? C++ and F don't do interface checking, they just toss a set of values from one to the other and hope for the best.

            1. chummels

              I added those flags just as an afterthought when things weren't working otherwise. The code fails the same way with or without those flags.

              And yes, I meant "aren't recognized" instead of "are recognized". Oops!

          2. Britton Smith author

            I'm seeing "molecular weight not converged" errors on kraken (though no seg fault) that are probably related. I will look into this and report back as soon as I can. Sorry everyone!

            1. chummels

              I get the "molecular weight not converged" errors (repeating into infinity) when actually doing the enzo grackle test run in the run directory. But when I just try to run the grackle example, that is when I get the errors above (e.g. NaN in edot...).

              1. Britton Smith author

                Hi everyone, I think I have figured out how to resolve this issue. It would appear that Matt Turk was correct and what needs to be added to the Grackle machine make files is the following:

                MACH_FFLAGS_REAL_64    = -fdefault-real-8 -fdefault-double-8
                

                This is because the fortran routines that were taken out of Enzo to become the Grackle predate Daniel Reynolds's precision fixes. I will update the source and documentation as soon as I have a free moment. I think I have this working with static libraries on NICS Kraken as well.

                1. chummels

                  So as I noted in a previous comment, those flags don't work for me on stampede. However, I found a related solution. By using the flags:

                  MACH_FFLAGS_INTEGER_64 = -i8
                  MACH_FFLAGS_REAL_64    = -r8
                  

                  I was able to get stampede to recognize the appropriate fortran precision and accept the C++ values correctly. Thanks for your insight, Matt Turk and your perseverance Britton Smith ! I'll PR that Makefile for stampede to the grackle repo, and I now approve this PR, since I can now run the AMRCosmology_Grackle test problem on an OS X (mountain lion) box and Stampede.

                  N.B. There was never anything wrong with this PR, it was just a problem with grackle's compilation itself.

    1. Britton Smith author

      The only thing I would like to happen before this gets accepted is for me to update the Grackle documentation and make files to avoid the type of errors we all saw. I will leave a comment in this PR when that happens. Does anyone else have any comments they would like to make before this goes in?

      1. Sam Skillman

        Britton Smith this is really great, and functionally I think this could be accepted and merged right away. The only remaining question I have comes from how the grackle compute_pressure, calculate_temperature, and compute_gamma compare to the methods in enzo itself. It seems like at first glance they might be equivalent, but could you comment on how you see enzo + grackle evolving if/when those methods diverge from each other? As I mentioned above, I think it may be useful to have a hangout so that we could talk about the API between enzo and grackle.

        1. Britton Smith author

          In the instances that are identical to the Grackle, I would eventually like to switch over to using the Grackle functions. However, in some of these, there are other cases that the Grackle does not explicitly cover. I think the best thing to do will be to keep the Enzo functions and have them call the Grackle functions inside or do the alternate calculation themselves. I would be interested in having a hangout to talk about the future of this. Is anyone other than Sam Skillman interested?

      2. Britton Smith author

        I just issued a PR to the main Grackle repo with the updates I wanted to make to the docs. This can be viewed here. This PR can now be pulled in if everyone is satisfied.