GetComponents --avoid-useless-downloads --propose-system-installs

[Boud Roukema]

WISHLIST: GetComponents --avoid-useless-downloads --propose-system-installs

AIMS: (1) increase likelihood of having wider-software-community-verified versions ("native" versions) of external libraries; (2) reduce download volume by about 40 percent or so; (3) reduce compile time;

ORIGINAL POST: http://cactuscode.org/pipermail/users/2018-September/003634.html

DESCRIPTION: In my recent (Sept 2018) download of Cactus, out of about 500 Mb total download, I have about 200Mb of downloaded external library source packages, most of which either (i) are superfluous because they already exist in my OS (Debian GNU/Linux), or (ii) should be superfluous, since they can more safely be installed using the apt/apt-get/aptitude/dpkg package management system. This proposal is to add an OS-dependent option to GetComponents that (i) checks if the OS is a debian derivative and if yes, then instead of downloading the source packages, extracts necessary versioning metadata from the OS, inserts this in the appropriate place in the Cactus system for defining -I include and -L library shell or makefile variables as appropriate; and/or (ii) checks if the OS is a debian derivative and if yes, proposes a list of system packages to install as root user that would avoid having to download and compile these. Proposed names for the options are (a) --avoid-useless-downloads and (b) --propose-system-installs.

(1) PREFER NATIVE software libraries

Even if --avoid-useless-downloads is not implemented, the point of --propose-system-installs is the overall software ecology. The Debian infrastructure will tend to lead to better checked scientific software in terms of portability, reproducibility, licence compatibility (reduce potential legal problems), security and generally scientific correctness (by following and contributing to upstream development and by encouraging unit tests and regression tests).

Several of the packages provided by my Sep 2018 ET install are older than the Debian stable versions, so a counterargument of "but the recommended Cactus source packages are more up-to-date" is invalid for (at least) my Sep 2018 install. Debian stable versions of packages are quite conservative in terms of security support and bug fixing.

As an individual user, the debian facilities for checking what packages my GR/cosmo software is dependent on will generally be more convenient and powerful than checking tar balls supplied in ET.

I have not checked if any of the ET ExternalLibraries have been customised for ET. For all the reasons listed above, it would generally be better if any customisations are proposed upstream, so that ET both benefits from and contributes to these libraries.

(2) DOWNLOAD VOLUME: http://cactuscode.org/pipermail/users/2018-September/003634.html Roland: http://cactuscode.org/pipermail/users/2019-March/003643.html and I seem to agree that the savings in terms of download would be about 30-40%. Given the high total volume, I don't think that this is negligible, though I accept that it has to be weighed against the effort required to set it up. This concerns --avoid-useless-downloads, but not --propose-system-installs. The latter would presumably invoke the former, but not vice-versa.

(3) reduce compile time;

Similarly to reducing download times, this could be considered a minor advantage of this proposal, since the makefiles should be intelligent enough not to force unnecessary recompiles. However, the initial compile duration is somewhat discouraging to people starting out on ET.

POLITER NAMES:

Politer names for --avoid-useless-downloads could be:

--avoid-redundant-downloads

--avoid-superfluous-downloads # but people are likely to misspell this

OUTLINE of script:

The idea for --propose-system-installs is not to try to do the install, since asking a user to give the root or sudo password seems to me unwise from a security point of view, given the whole idea of installing Cactus/ET in user space (by default). I'm only proposing that the perl script do something like output to the user (the following is pseudocode, I know very little perl):

# Find packages that ET supplies in `ExternalLibraries/` but are
# available for native installation
# (e.g. libfftw3-dev, liblapack-dev, libopenmpi-dev, openssl)

system_installable = $(...) # search in Cactus/arrangements/ExternalLibraries/

if(length($system_installable) > 0){
    print("Before doing a full download, you may wish to install the following\n
       packages instead of compiling from source, using");
    print(if $OS == debian_derivative ( "aptitude or
       apt-get" else "your package manager:\n"));
    print("$system_installable");
};

Options for some other common GNU/Linux and similar systems (GNU/kFreeBSD, Fedora, OSX) would be useful too.

I'm probably not the best person to propose a patch.