PyModules is an alternative implementation of the Environment Modules system for managing software environments on research computing clusters. It provides several new features:
- Uses simple, INI-style configuration files that share common values across multiple versions of a package.
- Module configuations are cached in a SQLite database, and editing/publishing are separate actions.
- Modules can be categorized. The
module availcommand lists by category or by fulltext search on the package name or version.
- Modules can be localized by CPU architecture, for either a specific vendor (AMD vs. Intel), SSE instruction set, or CPU model identifier.
module bincommand lists all binaries provided by a module.
PyModules is developed by the Center for Computation and Visualization at Brown University, and distributed freely for academic use under a non-commercial license (see the LICENSE file for full details).
Zip archives are available from the downloads page, or you can clone the Bitbucket repo for the latest development version.
For installation instructions, see the INSTALL file.
The system has included an auditing system to record the module loading/unloading by users. Although this system supported SQLite and file logging and has the related adapters, we have decided to hard-code the system to use MySQL, since the former two do not work well with concurrency.
When a user loads or unloads modules, the auditing system keeps track with the module name, module version, timestamp, and whether or not it is due to a global/user-defined module loading script. The actually logging work is done by a separate process so that the user won't be bothered.
By default, the system suppresses all the error messages and logs them to a failout
log file at user-directory (~/.modulelog.log). To debug the system, the suppression
should be released by editing
modulelog_config['suppress-error'] in modulecfg.py.
modulelog_config['suppress-error'] defines the timeout for the auditing
system to connect to the database.
This auditing system has three "red-buttons". Unsetting
in the environment disables the failout logging in case of read-only filesystems.
ENABLE_MODULELOG in the environment or setting
to False in modulecfg.py disables the entire logging system including the failout
logging and the actual module logging.
Howison M, Shen A, Loomis A. 2013. Building Software Environments for Research Computing Clusters. In Proceedings of the 27th Large Installation System Administration Conference (LISA '13), Washington, DC, USA. https://www.usenix.org/conference/lisa13/building-software-environments-research-computing-clusters
Please report any issues using the issue tracker at Bitbucket:
Hongkai (George) Sun