A few updates to the Makefiles. Biggest change was for the gnu compiler. The trick was to change the linking command so that it uses gfortran (mpif90) instead of the gnu C/C++ compiler. This meant an update to the linking flags.
A few quick questions:
I don't think we need three separate Makefiles for Stampede. I can see the use of the gnu and the intel, but is there a reason to keep around the vanilla one now?
Also, since you're using a slightly modified version of the Makefile that I generated for Stampede intel and sent you, could you add my name to the header of that file?
thanks for taking a look. I removed the vanilla file and added you as author
Looking at the description of your other PR for the grackle code with these same changes, you state:
"Both still require setting LD_LIBRARY_PATH to include the grackle lib dir. For some reason I also needed to add the HDF5 lib dir to this path when using the gnu compiler. This can be done using the stampede env variable TACC_HDF5_DIR."
So does one need to add the TACC_HDF5_DIR to one's LD_LIBRARY_PATH in order for this to work for gnu? If so, I think it should be noted in the Makefile somewhere, perhaps in the Machine Notes.
good point, updated the Machine notes for the gnu file.
Sorry about the delay. I'm trying to use your Makefiles here along with the Makefiles you've PRd to Grackle to make sure everything works as advertised. I've got intel working OK, but I'm having some problems with gnu. I'll write more about this soon. Once this passes, I think these are good to go.
Stampede is in maintenance mode today, so I have to wait until 7PM CST to work on it, but please do not accept until I double check this.
I'm logged into stampede now. It's not going down until Tues. Sept. 30th
You're right! Ack. Thanks, @galtay .
Tests all pass of course on my non-stampede testbot.
So I tested both Makefiles out on stampede along with the PR'd Makefiles for grackle. With the correction listed below, they both compile and work. When I run them on the quick test suite, they execute OK, but they fail to run the 12 FLD tests, which require the hypre module. Loading the hypre module does not enable this to work, so it may require that you manually specify the library and includes directory for hypre in the Makefile. Unfortunately, I cannot find those. Anyway, I think this is good enough for now, and you might mention in the Makefile comments that one must load and link hypre in order to get FLD to work OK.
Thanks for doing all the checks I should have done Cameron! There is a useful module command that shows you all the modifications a module makes. Executing module show hypre, I find the environment variable TACC_HYPRE_DIR. For now, I've added this variable to both Makefiles. I'm not sure how to run the quick test suite but I'm happy to try it.
Probably the easiest thing to do for the test suite is this. Once you've compiled enzo with gnu or intel or whatever, go into the enzo-dev/run directory and run this command:
python ./test_runner.py -o <output_dir>
where output_dir is some location you want enzo to store your simulation data. This command will automatically setup the quick test suite and run the simulations, it will try to run test comparisons against the old answer test files, which you have not generated, but that's OK. At the end, it will tell you which simulations ran and which did not. I was able to get all of the simulations to run except for the FLD ones because hypre wasn't working. I just loaded hypre with your updated makefile, and it still doesn't seem to work, so I don't know what is up. Again, I don't think this is a deal breaker, but it might be worth checking with others to see what they think on this (e.g. @drreynolds , @samskillman ) either for a potential solution, or if we can just go without FLD working here and accept the PR.
It looks like the HYPRE modules installed on Stampede are incompatible with the GNU compiler suite, since once the gcc module is loaded, the hypre module is unavailable. As a result, in order to run Enzo+FLD on Stampede, you cannot currently use the GNU compilers unless you want to build HYPRE yourself. A note to this effect should probably be added to the machine notes for this Makefile.
Once the intel module is loaded, there are two available hypre modules, hypre/2.9.1a-SmallScale (the default) and hypre/2.9.1a-LargeScale. According to the HYPRE folks, you should use the "LargeScale" version if you plan on running with over O(10000) MPI tasks; the difference is that "LargeScale" uses a slower but more scalable algorithm in setting up the linear solver. It would be useful if this were added to the machine notes for this Makefile, but it's not critical.
Now, in regard to compiling/linking, it looks like the existing setup that references the environment variable TACC_HYPRE_DIR works fine. I don't think that I've done anything special with my environment on Stampede. I just followed the instructions at the top of the Makefile on which modules to load, and everything linked correctly. I also ran a small test and it worked, so I imagine that other folks can use this Makefile as-is. Nice work!
I do have one comment, though. The use of LOCAL_GRACKLE_INSTALL = $(HOME)/code/grackle_intel and LOCAL_GRACKLE_INSTALL = $(HOME)/code/grackle_gnu in these two files is not at all helpful in a shared Makefile that is sent out with the Enzo source code, since everyone has different $(HOME) directories. If whoever's $(HOME)/code/grackle_intel and $(HOME)/code/grackle_gnu directories have the correct permissions, then I recommend that $(HOME) be replaced by the global path to that installation so that others can use these Makefiles without needing to install grackle locally in that exact location as well.
I think the implication is that grackle needs to be installed locally and then linked in your Makefile. I guess Gabe could make a globally-readable grackle directory, but I don't think it's expected of him to do that. I think just a note mentioning grackle linking may be sufficient.
Aside from that, this looks great. Thanks for your work on this @galtay !
Thanks for all the help folks. Re, the grackle path, I don't want other people's installs to depend on my local environment. Maybe it's best if I remove the path $(HOME)/code/grackle_intel (and the gnu version) and then put in the machine notes that if you'd like to use grackle you have to build a local copy and edit the variable LOCAL_GRACKLE_INSTALL.
You could do that, or you could leave the path and include the note in the machine notes.
I think at the end of this, these will be the most fully documented makefiles in the repo, which is great!
These are ready for merging whenever someone has the time.