This PR adds support for the Grackle chemistry and cooling library to Enzo. Use of Grackle requires compiling with "make grackle-yes". All Grackle functionality should be inside #ifdef USE_GRACKLE, but please leave a comment if you find anything that is not.
As I mentioned in my email to enzo-dev a while ago, Grackle is a chemistry and cooling library that was originally derived from Enzo's MultiSpecies and Cloudy cooling modules. However, there have been a number of updates to the UV background and metal cooling functionality make this not only unique from the original implementation, but potentially a more attractive option for simulations without radiative transfer, such as:
UV backgrounds are treated via interpolating from data tables loaded from disk rather
than piece-wise polynomial functions, making it somewhat easier to add support for
new background models.
UV background and cooling data are contained within the same input file and are
more consistent than the currently available Cloudy cooling data and Enzo UV
background models. Currently, the Grackle distribution comes with UV background
and cooling data for two different models:
Unlike the original Cloudy cooling which required separate input files for cooling
before the UV background turns on and after, all data is contained in a single
table. This means one no longer has to run with one input file to the redshift
where the UV background starts, stop the simulation, and restart with another input
Also unlike the original Cloudy cooling module, Grackle supports the option to also
solve the primordial cooling via interpolation from a table. Thus, one is no longer
required to run with the MultiSpecies functionality in order to calculate the
primordial component. This simplified method is somewhat faster and requires fewer
baryon fields to be stored, lowering the ram and disk footprint.
This also enables the use of Enzo for the AGORA project with the required common physics package, a.k.a. Grackle.
What has been added:
support to solve chemistry cooling and calculate cooling time using Grackle functions
narrative documentation discussing Grackle in general, how to build Enzo with it, and a full description of parameters
sample cosmology simulation parameter file using Grackle in run/CosmologySimulation/AMRCosmology_Grackle
I ran a very small cosmological simulation using Cloudy cooling, Grackle with MultiSpecies 1, and Grackle with MultiSpecies 0 (fully-tabulated). Note, the input data are not the same for the Cloudy run and the Grackle runs. The Grackle data corresponds to a newer version of the Haardt & Madau background which starts at a higher redshift (z = 14.849 instead of z = 8.9). Below are projections of metallicity and phase plots of cell mass in bins of density and temperature for each of the three simulations. The strange centering in the projections is due to the fact that this is a pretty low-resolution nested simulation and we are viewing only the refinement region.
Grackle w/ MultiSpecies = 1
Grackle Fully-tabulated (note: phase plot y range is different)
The most notable different is the addition of a small amount of cold gas at roughly 1e3 K in the Grackle runs. I'm not entirely sure where exactly this comes from, but I don't think it is indicative of something incorrect.
Do you think it would be possible to turn on grackle using a runtime parameter rather than a compile-time parameter?
You can turn grackle on with the use_grackle parameter. However, I wanted to make a compile time option as well so as to not force people to install grackle if they never intended to use it.
Makes sense. For some reason I was thinking you could do this outside the preprocessor but clearly everyone would need a grackle.h file for that to work.
In the future we may want to come back to this and think about making grackle a required dependency, particularly if grackle starts to become more popular than enzo's built-in cooling routines.
I've tested this out, and once I actually follow the directions it works great! Thanks for all of this work, @brittonsmith . Would you be up for a G+ dev hangout in the near future to have folks give feedback on the structure of enzo <--> grackle?
First, let me say that I'm very excited that you're PR'ing this, as I'm eager to use it for my own sims. Thank you for your hard work!
I followed your installation instruction and was able to get grackle installed on TACC stampede without any problems. I then followed your instructions for installing and building grackle support in enzo. This was pretty straightforward too. Finally, I tried to run your AMRCosmology Test simulation in the run directory, but I'm getting failures. Essentially, it builds the ICs fine, runs ring ok, and then starts up the simulation, but it keeps repeating this line in STDERR:
"Mean molecular weight not converged! NaN NaN"
I found this in the Grackle Source for cool1d_multi_g.F, in that it cannot seem to converge on Mu when adding in the different metal species. But I'm not exactly sure why this is happening. I'm just running with all of your defaults. It looks like it gets past all of the initializations and is actually starting calculation to cause this error. Any ideas?
@samskillman : were you able to run these test runs OK?
To follow up on this comment, I was able to get enzo-grackle installed and working on a Darwin box (mountain lion). Everything seemed to work fine after I made the environment variable modification noted below so that enzo could find the grackle libraries. But I still don't know what's going awry with grackle-enzo on Stampede as discussed above.
@samskillman mentioned to me he had the same problem and I think he said that he fixed it by making sure he was using a separate Makefile for grackle and enzo. @samskillman, can you comment on this?
Sadly, I did have this error and managed to fix it (user error on my end), but I don't remember precisely what it was. I believe the key parts for me were:
Don't just copy the Make.mach.whatever from my .enzo/ to .grackle/ . I think this is mostly because of the -fPIC and it didn't have the necessary install directory flags.
Make sure to set the MACH_INSTALL_PREFIX to somewhere such that $(MACH_INSTALL_PREFIX)/lib is already in your LD_LIBRARY_PATH (DYLD for silly macs) or that you later add to your (DY)LD_LIBRARY_PATH.
Make sure you actually do "make install" from grackle/src and not just "make"
@chummels has correctly pointed out to me that the grackle documentation states that you can use your Enzo make file to build grackle, which is not true and seems to be the cause of these problems. I will correct this as soon as I can.
I'm seeing the same issues as Cameron on Stampede with an independently generated makefile for grackle. No idea what's going wrong. Let me know if it would help to see my makefiles.
Yeah, I went back and generated a brand new Makefile on Stampede by modifying Make.mach.unknown. It too fails. The easiest way to demonstrate this is to (from grackle src/clib dir):
Using gdb with this, it appears the error occurs when c++ passes pointers to fortran programs, specifically in the call to solve_rate_cool_g.F, many of the values it gets passed are recognized, which yield a quick segfault. I'm not sure how to fix this--perhaps some additional compiler flag in the Makefile? I'm sort of out of ideas here.
My guess is that it's related to -fdefault-real-8 and -fdefault-double-8 in your makefile. Try seeing if it fails in 32 bit mode. Also, did you mean "many of the values it gets passed aren't recognized"? C++ and F don't do interface checking, they just toss a set of values from one to the other and hope for the best.
I added those flags just as an afterthought when things weren't working otherwise. The code fails the same way with or without those flags.
And yes, I meant "aren't recognized" instead of "are recognized". Oops!
I'm seeing "molecular weight not converged" errors on kraken (though no seg fault) that are probably related. I will look into this and report back as soon as I can. Sorry everyone!
I get the "molecular weight not converged" errors (repeating into infinity) when actually doing the enzo grackle test run in the run directory. But when I just try to run the grackle example, that is when I get the errors above (e.g. NaN in edot...).
Hi everyone, I think I have figured out how to resolve this issue. It would appear that @MatthewTurk was correct and what needs to be added to the Grackle machine make files is the following:
This is because the fortran routines that were taken out of Enzo to become the Grackle predate @drreynolds's precision fixes. I will update the source and documentation as soon as I have a free moment. I think I have this working with static libraries on NICS Kraken as well.
So as I noted in a previous comment, those flags don't work for me on stampede. However, I found a related solution. By using the flags:
I was able to get stampede to recognize the appropriate fortran precision and accept the C++ values correctly. Thanks for your insight, @MatthewTurk and your perseverance @brittonsmith ! I'll PR that Makefile for stampede to the grackle repo, and I now approve this PR, since I can now run the AMRCosmology_Grackle test problem on an OS X (mountain lion) box and Stampede.
N.B. There was never anything wrong with this PR, it was just a problem with grackle's compilation itself.
Is there any reason why this should not be merged?
The only thing I would like to happen before this gets accepted is for me to update the Grackle documentation and make files to avoid the type of errors we all saw. I will leave a comment in this PR when that happens. Does anyone else have any comments they would like to make before this goes in?
@brittonsmith this is really great, and functionally I think this could be accepted and merged right away. The only remaining question I have comes from how the grackle compute_pressure, calculate_temperature, and compute_gamma compare to the methods in enzo itself. It seems like at first glance they might be equivalent, but could you comment on how you see enzo + grackle evolving if/when those methods diverge from each other? As I mentioned above, I think it may be useful to have a hangout so that we could talk about the API between enzo and grackle.
In the instances that are identical to the Grackle, I would eventually like to switch over to using the Grackle functions. However, in some of these, there are other cases that the Grackle does not explicitly cover. I think the best thing to do will be to keep the Enzo functions and have them call the Grackle functions inside or do the alternate calculation themselves. I would be interested in having a hangout to talk about the future of this. Is anyone other than @samskillman interested?
I just issued a PR to the main Grackle repo with the updates I wanted to make to the docs. This can be viewed here. This PR can now be pulled in if everyone is satisfied.
I'm A-OK with this, but we should wait to see if @samskillman or anyone else have objections. Again, great work, @brittonsmith .