Want stand-alone parameter parser

[Erik Schnetter]

I would like to have an easy-to-use stand-alone parameter file parser. Presumably, this utility would be build as a Cactus utility in the flesh. I would take a parameter file as input, and would output parameter values as output (probably either in some ASCII format or in xml). It would also know about default values of parameters.

This utility could answer the following questions: - which parameters are set in a parameter file? - what is the value of parameter X (which may have its default value)? - list all parameters that are set in the parameter file - list all parameters and their values

Bonus points if it also knows about which thorns are active; this is non-trivial since some thorns are activated automatically.

Keyword:

[Ian Hinder]

The output depends on which thorns have been included in a particular configuration, so a utility compiled with this knowledge within a configuration seems the correct approach. In this sense, I don't think this could be considered "standalone"; it would be tied to the configuration which was used to build it. Why not use the Cactus executable, which already has the knowledge of what parameters are set, and the default values etc?

With that in mind, how about adding a command line option to Cactus to print the values of all parameters and then exit. It already had options to print the schedule tree and the list of compiled thorns. It seems natural to also be able to print the values of all the parameters. It will automatically satisfy the bonus point part as well.

[Roland Haas]

The Cactus executable might be a bad choice. There are machines (the Crays) where you cannot even execute the main executable on the head node. So a separate utility is likely required.

[Ian Hinder]

I thought about this, but why is there a difference between the main executable and the utilities? Is it because the main executable calls MPI_Init? This could be bypassed for printing the parameters. Or is it that the main executable is cross-compiled whereas the utilities are not?

[Ian Hinder]

Erik, why do you want a parameter file parser? I currently parse the parameter file to get certain information about a simulation when I am analysing the data. However, this is not really sufficient. I don't generally have the Cactus tree which was used for the simulation, and defaults and the interpretation of parameters may have changed. Similarly, I have to re-implement the logic that Cactus uses to interpret the high-level parameters into low-level results. For my purposes, it would be better to have a more definitive set of metadata output by the simulation itself, which would include any derived values from the parameters, and maybe also computed physics quantities (See #551 for one idea of this). Much of my parameter parsing is related to finding information about the grid structure. Some of this information would already be available if you set the right parameters, but it would be helpful to have just one parameter which enabled output of a single file which contained all the simulation metadata, including the grid structure, in a format which was easy to parse. Since some things might change with time, and we would have to work out if we want this to be reflected in the metadata file.

[Erik Schnetter]

I want a parameter parser so that I know the parameters and their values, given a parameter file.

I assume that using the Cactus executable may work. It's a bit clunky to use (since it may be much larger than a utility) and/or may require mpirun, but this can be dealt with. (We can either have a "runcactus" script that remembers which mpirun to use, or can use Simfactory's "sim run" command, or use a similar solution.) I also don't like that this may require going through a queuing system when cross-compiling, but in this case, I would assume that one can instead hopefully use an identical source tree on a different machine.

[Ian Hinder]

I'm still not quite sure what you want to use this for, so I can't see what aspects are important. Given just a parameter file, it is not possible to know the parameters and their values, because you need the Cactus source and the thornlist to determine the default values. Do you need this feature just for your own simulations? If so, you can set IO::parfile_write = "generate", in which case you get all parameters (not just those explicitly set) in the parameter file written to the output directory. If you (as I) want to know the values of parameters from simulation output, then you won't necessarily have the Cactus tree and thornlist, which is why I wrote comment:4. If you want to know the parameters for a parameter file when you do still have the source tree etc, a utility or the Cactus executable would be suitable. For this, you would need to be examining the parameter file on a machine where the Cactus executable or utility has been compiled with the correct source tree. I suggest that the code in Cactus which parses the parameter file should not be duplicated but should be reused. Given the discussions about adding high-level features to the parameter file, and the fact that you say you want to know the parameters and their values, a "parser" is not sufficient. You also need something which interprets the parse tree and evaluates expressions etc. If the code in Cactus which performs all this logic (and which reads the CCL files etc) is modular enough, then maybe it could be compiled into a utility, but I doubt that this is the case. Why do we need mpirun to run the cactus executable? Can we not check the command line arguments and then just not call mpi_init if we only want the parameters? The only time I can imagine not being able to run the executable would be if it was cross-compiled.

[Wolfgang Kastaun]

I agree with Ian. If this is intended to get information about existing simulations, I think a parameter file parser would be just a workaround for the lack of metadata stored by Cactus. We should separate two functionalities: How to specify parameters, i.e. input parfile format, and how to get parameters from existing simulations in postprocessing scripts. It is difficult to satisfy those two needs with a single parameter file format. For my postprocessing tasks the following points are important:

1. A file with all parameters, including those set from default values, should be created when running a simulation.
2. In contrast to the existing generated.par, it should be machiene readable, and it should contain information on the type of the parameters (int, bool, string etc) as well.
3. A possibility to re-create this file from an available Cactus tree without running the simulation. I favor this function to be build into the executable because sometimes the Cactus tree is gone, but the executable is still stored in the simulation folder.

This way, new simulations can be postprocessed in the future without having the executable. For the format, I suggest xml or hdf5. Both is easily readable from most languages including Python.

[Erik Schnetter]

Regarding MPI_Init: It is not possible to circumvent calling MPI_Init, as this is the first thing that needs to be done in an executable. There is no way to make a decision before this point, since e.g. command line arguments and environment variables are not yet available.

Regarding outputting parameters: You can set IO::parfile_write="generate", which outputs the current values instead of copying the parameter file. Thorn Formaline also outputs all parameter values in a format that is easier to parse, but people haven't really been interested.

A while ago, some people mentioned they would parse parameter files themselves and thus didn't like a complex parameter file syntax. Instead of parsing parameter files, there could be a generic parameter file parser that is easy to use and gets the syntax correct. Now I see there is much opposition to such an idea -- we should remember this next time we discuss parameter file syntax, as people apparently aren't supposed to parse parameter files by themselves anyway.