HTTPS SSH

PyModules is an alternative implementation of the
Environment Modules system
for managing software environments on research computing clusters.
It provides several new features:

  • Uses simple, INI-style configuration files that share common values across
    multiple versions of a package.
  • Module configuations are cached in a SQLite database, and editing/publishing
    are separate actions.
  • Modules can be categorized. The module avail command lists by
    category or by fulltext search on the package name or version.
  • Modules can be localized by CPU architecture, for either a specific vendor
    (AMD vs. Intel), SSE instruction set, or CPU model identifier.
  • New module bin command lists all binaries provided by a module.

PyModules is developed by the
Center for Computation and Visualization
at Brown University, and distributed freely for academic use under a
non-commercial license (see the LICENSE file for full details).

Installation

Zip archives are available from the downloads
page
, or you can clone the
Bitbucket repo for the latest development version.

For installation instructions, see the INSTALL file.

Auditing

The system has included an auditing system to record the module loading/unloading
by users. Although this system supported SQLite and file logging and has the related
adapters, we have decided to hard-code the system to use MySQL, since the former two
do not work well with concurrency.

When a user loads or unloads modules, the auditing system keeps track with the module
name, module version, timestamp, and whether or not it is due to a global/user-defined
module loading script. The actually logging work is done by a separate process so that
the user won't be bothered.

By default, the system suppresses all the error messages and logs them to a failout
log file at user-directory (~/.modulelog.log). To debug the system, the suppression
should be released by editing modulelog_config['suppress-error'] in modulecfg.py.
Meanwhile, modulelog_config['suppress-error'] defines the timeout for the auditing
system to connect to the database.

This auditing system has three "red-buttons". Unsetting ENABLE_MODULELOGFAILOUT
in the environment disables the failout logging in case of read-only filesystems.
Unsetting ENABLE_MODULELOG in the environment or setting modulelog_config['enabled']
to False in modulecfg.py disables the entire logging system including the failout
logging and the actual module logging.

Documentation

See http://ccv.brown.edu/mhowison/pymodules

Citing

Howison M, Shen A, Loomis A. 2013. Building Software Environments for Research
Computing Clusters. In Proceedings of the 27th Large Installation System
Administration Conference (LISA '13), Washington, DC, USA.
https://www.usenix.org/conference/lisa13/building-software-environments-research-computing-clusters

Issues

Please report any issues using the issue tracker at Bitbucket:

https://bitbucket.org/mhowison/pymodules/issues

Authors

Mark Howison
Andy Loomis
Aaron Shen
Hongkai (George) Sun