Matthew Turk committed 0a9ecad

Initial libconfig import

Comments (0)

Files changed (1)


+.. _libconfig-proposal:
+A Modern Configuration System
+.. sectionauthor:: Matthew Turk <>
+This proposal aims to revamp the Enzo configuration system to ensure that it is
+modular, extensible, and easier to read.
+Current State of the Code
+Currently Enzo outputs a huge number of parameters.  At last count, the output
+parameter file is over 500 lines, in a monolithic file.  Most of these are set
+to the default value in the code (a behavior I do not think should change).
+Adding a new parameter is complicated, and requires touching the code in four
+  * ``ReadParameterFile.C``
+  * ``WriteParameterFile.C``
+  * ``SetDefaultGlobalValues.C``
+  * ``global_data.h``
+Furthermore, the addition of a single parameter touches ``global_data.h``,
+which requires recompilation of the entire code base, even when only adding
+a parameter used in a single place.
+Additionally, reading in parameters is hugely painful.  To read in a list,
+specialized routines must be called.  Reading strings is annoying.  All in all,
+it was a very good system that was straightforward to use, required nothing
+external, and was incredibly reliable for a long time.  But unfortunately, it
+has not scaled with the complexity of Enzo.
+Proposed Revisions
+I believe that we should utilize the commodity library `libconfig
+<>` for reading and writing parameter
+This would eliminate the existing Enzo parameter files.  I believe we should
+separate our parameters into (formal, rather than de facto) categories, write
+them to disk in that form, and read them from disk in that form.
+libconfig works by parsing the entire parameter file at once, either into a
+clean set of hierarchical options or into an existing set of hierarchical
+options.  This essentially means that the parameter files would look something
+like this, although the specific hierarchy can be decided upon later:
+.. code-block:: none
+   SimulationParameters = {
+        HydroMethod = "PPM";
+        RefinementCriteria = (2, 4, 6);
+        JeansParameterCriteria = {
+            SafetyFactor = 16;
+            CoolTemperature = 200;
+        }
+   };
+This is an example only showing a simple example.  When read in, these
+parameters can be accessed very easily:
+.. code-block:: c
+   JeansSafety = config_setting_get_int(config_obj,
+        "SimulationParameters.JeansParameterCriteria.SafetyFactor");
+There is some boilerplate code for reading the config file in initially.
+A few additional steps must also be taken to read in lists, but these steps
+can also provide back to the code the length of a given list of values.
+Strings are returned as char*'s that are managed by libconfig, not by Enzo.
+We would supply a default, in-memory version of the parameter file to
+initialize default variables, and then all values would be read back out.
+In a longer reaching plan, in a system where Enzo contained several event
+hooks, these hooks can be controlled by the parameter file:
+.. code-block:: none
+   BottomOfHierarchy = ("CallPython", "OutputOnDensity");
+   TopOfHierarchy = ("CallPython", "PrintStatistics");
+One could imagine reading in the array of strings for ``BottomOfHierarchy`` and
+then executing each of the named routines in sequence.
+This would touch the code in a few places.  Because it is provided through a
+formal grammar, usage of the libconfig parameter file mechanism would
+completely eliminate the need to have ``ReadParameterFile`` and
+``WriteParameterFile`` be as long, complex, and difficult to handle as they
+currently are.
+Furthermore, the conceptual separation of individual elements of a
+configuration file would be much clearer.
+Finally, we are now able to utilize strings to identify modules in a simple,
+straightforward and clear fashion.  This will prevent an immense number of
+If taken to its fullest extent, this will also provide the ability to
+completely remove global variables and only pass around a single "config"
+The benefits for the developers will be broad, as it will enable a much simpler
+interface to an on-disk parameter file.  The benefits for users will be
+similar, as parameter files will be conceptually separated into manageable
+Stages of Changes
+There are several phases of possible changes, and how deep we wish to go
+depends highly on how the discussion around :ref:`event-handling` proceeds.
+I have already created a version of Enzo that includes ``libconfig`` in its
+build process.  ``libconfig`` is licensed LGPL, which is compatible with the
+Enzo license.  Changes to Enzo itself are still governed by the modified BSD
+license, but changes to ``libconfig`` are governed by the LGPL, which requires
+making available the source if the binary is provided to end-users.  This
+primarily affects distribution, and should be considered equivalent to linking
+against most commodity libraries such as those provided by the GNU toolchain.
+It does not "infect" Enzo.  (Strictly speaking, I am not opposed to it
+infecting Enzo, but it does not.)
+ #. Decide on a hierarchical separation of parameters
+ #. Create a new fiducial parameter file, using only default values
+ #. Implement the read / write operations
+ #. Implement converters for old versions of the code
+ #. Insert hooks into event handling into the configuration mechanism.
+Compatibility Issues
+This would be a forward-facing, incompatible change to Enzo.  A converter could
+be written to convert old parameter files to the new format.  I believe that
+naming parameter files *written by* Enzo ``.enzo`` or something similar would
+assist with this.
+However, once a change like this has been made, it can provide a much easier
+forward-compatibility path.  We will only need to move to libconfig once, and
+we can change the internal data structures (such as ``global_data.h``) many
+Work Suggestions
+There are three clear working groups, with the first two operating in parallel
+and the third coming later:
+ #. Deciding on a conceptual separation of parameters, as well as removing old
+    parameters.  Evaluate which parameters would be better off as strings
+    instead of integers or toggles.  Evaluate creating mutually exclusive
+    parameter settings for module incompatibility.
+ #. Implement new versions of ``ReadParameterFile.C`` and
+    ``WriteParameterFile.C`` that utilize libconfig.
+ #. Write a converter for old parameter files to the new format.
+Comments and Votes
+.. toctree::
+   :maxdepth 1:
+   comments_*