enzo-workshop-proposals / libconfig / index.rst

A Modern Configuration System

This proposal aims to revamp the Enzo configuration system to ensure that it is modular, extensible, and easier to read.

Current State of the Code

Currently Enzo outputs a huge number of parameters. At last count, the output parameter file is over 500 lines, in a monolithic file. Most of these are set to the default value in the code (a behavior I do not think should change).

Adding a new parameter is complicated, and requires touching the code in four places:

  • ReadParameterFile.C
  • WriteParameterFile.C
  • SetDefaultGlobalValues.C
  • global_data.h

Furthermore, the addition of a single parameter touches global_data.h, which requires recompilation of the entire code base, even when only adding a parameter used in a single place.

Additionally, reading in parameters is hugely painful. To read in a list, specialized routines must be called. Reading strings is annoying. All in all, it was a very good system that was straightforward to use, required nothing external, and was incredibly reliable for a long time. But unfortunately, it has not scaled with the complexity of Enzo.

Proposed Revisions

I believe that we should utilize the commodity library libconfig <http://www.hyperrealm.com/libconfig/> for reading and writing parameter files.

This would eliminate the existing Enzo parameter files. I believe we should separate our parameters into (formal, rather than de facto) categories, write them to disk in that form, and read them from disk in that form.

libconfig works by parsing the entire parameter file at once, either into a clean set of hierarchical options or into an existing set of hierarchical options. This essentially means that the parameter files would look something like this, although the specific hierarchy can be decided upon later:

This is an example only showing a simple example. When read in, these parameters can be accessed very easily:

JeansSafety = config_setting_get_int(config_obj,
     "SimulationParameters.JeansParameterCriteria.SafetyFactor");

There is some boilerplate code for reading the config file in initially. A few additional steps must also be taken to read in lists, but these steps can also provide back to the code the length of a given list of values. Strings are returned as char*'s that are managed by libconfig, not by Enzo. We would supply a default, in-memory version of the parameter file to initialize default variables, and then all values would be read back out.

In a longer reaching plan, in a system where Enzo contained several event hooks, these hooks can be controlled by the parameter file:

One could imagine reading in the array of strings for BottomOfHierarchy and then executing each of the named routines in sequence.

Benefits

This would touch the code in a few places. Because it is provided through a formal grammar, usage of the libconfig parameter file mechanism would completely eliminate the need to have ReadParameterFile and WriteParameterFile be as long, complex, and difficult to handle as they currently are.

Furthermore, the conceptual separation of individual elements of a configuration file would be much clearer.

Finally, we are now able to utilize strings to identify modules in a simple, straightforward and clear fashion. This will prevent an immense number of collisions.

If taken to its fullest extent, this will also provide the ability to completely remove global variables and only pass around a single "config" object.

The benefits for the developers will be broad, as it will enable a much simpler interface to an on-disk parameter file. The benefits for users will be similar, as parameter files will be conceptually separated into manageable chunks.

Stages of Changes

There are several phases of possible changes, and how deep we wish to go depends highly on how the discussion around :ref:`event-handling` proceeds.

I have already created a version of Enzo that includes libconfig in its build process. libconfig is licensed LGPL, which is compatible with the Enzo license. Changes to Enzo itself are still governed by the modified BSD license, but changes to libconfig are governed by the LGPL, which requires making available the source if the binary is provided to end-users. This primarily affects distribution, and should be considered equivalent to linking against most commodity libraries such as those provided by the GNU toolchain. It does not "infect" Enzo. (Strictly speaking, I am not opposed to it infecting Enzo, but it does not.)

  1. Decide on a hierarchical separation of parameters
  2. Create a new fiducial parameter file, using only default values
  3. Implement the read / write operations
  4. Implement converters for old versions of the code
  5. Insert hooks into event handling into the configuration mechanism.

Compatibility Issues

This would be a forward-facing, incompatible change to Enzo. A converter could be written to convert old parameter files to the new format. I believe that naming parameter files written by Enzo .enzo or something similar would assist with this.

However, once a change like this has been made, it can provide a much easier forward-compatibility path. We will only need to move to libconfig once, and we can change the internal data structures (such as global_data.h) many times.

Work Suggestions

There are three clear working groups, with the first two operating in parallel and the third coming later:

  1. Deciding on a conceptual separation of parameters, as well as removing old parameters. Evaluate which parameters would be better off as strings instead of integers or toggles. Evaluate creating mutually exclusive parameter settings for module incompatibility.
  2. Implement new versions of ReadParameterFile.C and WriteParameterFile.C that utilize libconfig.
  3. Write a converter for old parameter files to the new format.

Comments and Votes

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.