HTTPS SSH

PyModules is an alternative implementation of the Environment Modules system for managing software environments on research computing clusters. It provides several new features:

  • Uses simple, INI-style configuration files that share common values across multiple versions of a package.
  • Module configuations are cached in a SQLite database, and editing/publishing are separate actions.
  • Modules can be categorized. The module avail command lists by category or by fulltext search on the package name or version.
  • Modules can be localized by CPU architecture, for either a specific vendor (AMD vs. Intel), SSE instruction set, or CPU model identifier.
  • New module bin command lists all binaries provided by a module.

PyModules is developed by the Center for Computation and Visualization at Brown University, and distributed freely for academic use under a non-commercial license (see the LICENSE file for full details).

Installation

Zip archives are available from the downloads page, or you can clone the Bitbucket repo for the latest development version.

For installation instructions, see the INSTALL file.

Auditing

The system has included an auditing system to record the module loading/unloading by users. Although this system supported SQLite and file logging and has the related adapters, we have decided to hard-code the system to use MySQL, since the former two do not work well with concurrency.

When a user loads or unloads modules, the auditing system keeps track with the module name, module version, timestamp, and whether or not it is due to a global/user-defined module loading script. The actually logging work is done by a separate process so that the user won't be bothered.

By default, the system suppresses all the error messages and logs them to a failout log file at user-directory (~/.modulelog.log). To debug the system, the suppression should be released by editing modulelog_config['suppress-error'] in modulecfg.py. Meanwhile, modulelog_config['suppress-error'] defines the timeout for the auditing system to connect to the database.

This auditing system has three "red-buttons". Unsetting ENABLE_MODULELOGFAILOUT in the environment disables the failout logging in case of read-only filesystems. Unsetting ENABLE_MODULELOG in the environment or setting modulelog_config['enabled'] to False in modulecfg.py disables the entire logging system including the failout logging and the actual module logging.

Documentation

See http://ccv.brown.edu/mhowison/pymodules

Citing

Howison M, Shen A, Loomis A. 2013. Building Software Environments for Research Computing Clusters. In Proceedings of the 27th Large Installation System Administration Conference (LISA '13), Washington, DC, USA. https://www.usenix.org/conference/lisa13/building-software-environments-research-computing-clusters

Issues

Please report any issues using the issue tracker at Bitbucket:

https://bitbucket.org/mhowison/pymodules/issues

Authors

Mark Howison
Andy Loomis
Aaron Shen
Hongkai (George) Sun