GetComponents --avoid-useless-downloads --propose-system-installs

Issue #2239 new
Boud Roukema created an issue

WISHLIST: GetComponents --avoid-useless-downloads --propose-system-installs

AIMS: (1) increase likelihood of having
wider-software-community-verified versions ("native" versions) of
external libraries; (2) reduce download volume by about 40 percent or
so; (3) reduce compile time;

ORIGINAL POST:
http://cactuscode.org/pipermail/users/2018-September/003634.html

DESCRIPTION:
In my recent (Sept 2018) download of Cactus, out of about 500 Mb total
download, I have about 200Mb of downloaded external library source
packages, most of which either (i) are superfluous because they already
exist in my OS (Debian GNU/Linux), or (ii) should be superfluous,
since they can more safely be installed using the
apt/apt-get/aptitude/dpkg package management system. This proposal
is to add an OS-dependent option to GetComponents that (i) checks if the
OS is a debian derivative and if yes, then instead of downloading the
source packages, extracts necessary versioning metadata from the OS,
inserts this in the appropriate place in the Cactus system for defining
-I include and -L library shell or makefile variables as appropriate;
and/or (ii) checks if the OS is a debian derivative and if yes, proposes
a list of system packages to install as root user that would avoid
having to download and compile these. Proposed names for the options
are (a) --avoid-useless-downloads and (b) --propose-system-installs.

(1) PREFER NATIVE software libraries

Even if --avoid-useless-downloads is not implemented,
the point of --propose-system-installs is the overall software
ecology. The Debian infrastructure will tend to lead to better checked
scientific software in terms of portability, reproducibility,
licence compatibility
(reduce potential legal problems),
security and generally scientific correctness (by following
and contributing to upstream development and by encouraging unit
tests and regression tests).

Several of the packages provided by my Sep 2018 ET install are older than
the Debian stable versions, so a counterargument of "but the
recommended Cactus source packages are more up-to-date" is invalid
for (at least) my Sep 2018 install. Debian stable versions of packages are
quite conservative in terms of security support and bug fixing.

As an individual user, the debian facilities for checking what packages
my GR/cosmo software is dependent on will generally be more convenient
and powerful than checking tar balls supplied in ET.

I have not checked if any of the ET ExternalLibraries have been
customised for ET. For all the reasons listed above, it would generally
be better if any customisations are proposed upstream, so that ET
both benefits from and contributes to these libraries.

(2) DOWNLOAD VOLUME:
http://cactuscode.org/pipermail/users/2018-September/003634.html
Roland: http://cactuscode.org/pipermail/users/2019-March/003643.html
and I seem to agree that the savings in terms of download would be
about 30-40%. Given the high total volume, I don't think that this is
negligible, though I accept that it has to be weighed against the
effort required to set it up. This concerns --avoid-useless-downloads,
but not --propose-system-installs. The latter would presumably invoke
the former, but not vice-versa.

(3) reduce compile time;

Similarly to reducing download times, this could be considered a minor
advantage of this proposal, since the makefiles should be intelligent
enough not to force unnecessary recompiles. However, the initial compile
duration is somewhat discouraging to people starting out on ET.

POLITER NAMES:

Politer names for --avoid-useless-downloads could be:

--avoid-redundant-downloads

--avoid-superfluous-downloads # but people are likely to misspell this

OUTLINE of script:

The idea for --propose-system-installs is not to try to do the
install, since asking a user to give the root or sudo password seems
to me unwise from a security point of view, given the whole idea of
installing Cactus/ET in user space (by default). I'm only proposing that
the perl script do something like output to the user (the following
is pseudocode, I know very little perl):

# Find packages that ET supplies in `ExternalLibraries/` but are
# available for native installation
# (e.g. libfftw3-dev, liblapack-dev, libopenmpi-dev, openssl)

system_installable = $(...) # search in Cactus/arrangements/ExternalLibraries/

if(length($system_installable) > 0){
    print("Before doing a full download, you may wish to install the following\n
       packages instead of compiling from source, using");
    print(if $OS == debian_derivative ( "aptitude or
       apt-get" else "your package manager:\n"));
    print("$system_installable");
};

Options for some other common GNU/Linux and similar systems (GNU/kFreeBSD,
Fedora, OSX) would be useful too.

I'm probably not the best person to propose a patch.

Comments (1)

  1. Log in to comment