Thornlists entry for external libraries on github is broken

[Jonah Miller]

The entry for external libraries in the thornlist looks like this:

‌

# Additional Cactus thorns
!TARGET   = $ARR
!TYPE     = svn
!URL      = https://github.com/EinsteinToolkit/$1-$2.git/trunk

‌

this is a git repo, not an svn repo. So the entry should be

‌

# Additional Cactus thorns
!TARGET   = $ARR
!TYPE     = git
!URL      = https://github.com/EinsteinToolkit/$1-$2.git

I’m happy to submit a PR. But I wanted to report the bug first, since I might not get to it today. This appears to be present in both the development version and Proca.

[Roland Haas]

This is intentional (and should be working, do you observe a failure while checking out the code?). GitHub supports accessing the (git) repositories via subversion. In the ET we use this to only download the (large) tarballs for the current release of the ExternalLibrary and not all tarballs stored in history. See https://help.github.com/en/articles/support-for-subversion-clients for GitHub’s doc. I cannot find a ticket about the change right now, but it was discussed in some of the ET calls so should be in the minutes (somewhere…).

If you stumbled across this because GitHub refused to talk to your cluster’s subversion client, then the reason for the failure is most likely a too old OpenSSL library on your cluster.

[Jonah Miller]

Ah, I see! Yes, I discovered this when github refused to talk to my machine. Okay, I’ll mark this as resolved.

It might be worth mentioning on the downloads page that the user needs an up-to-date version of OpenSSL. And/or provide an alternative version of the thornlist for those with out-of-date libraries.

[Jonah Miller]

Not a bug.

[Roland Haas]

The notice can be added but I do not know of a workaround that I could provide (other than: download on your laptop then use simfactory/bin/sim sync to copy to the cluster). Since the OpenSSL library itself is to blame, switching from svn to git does not fix the issue, unless of course your git installation (eg via an environment module) links to a newer version of openssl than your svn executable (which may be part of the OS packages). The latter is the case on BW by default.

I tried switching https:// to http:// but github just redirects that back to https://.
GitHub’s announcement (https://github.blog/2018-02-01-crypto-removal-notice/) does not say but apparently tlsv1.2 support is in OpenSSL since 2012 (https://stackoverflow.com/questions/48178052/when-was-tls-1-2-support-added-to-openssl) which is unfortunately just “recent” enough that certain clusters with frozen software stacks (such as Blue Waters) are affected.

[Jonah Miller]

The failure I encountered was actually on my Apple laptop, and switching to git did fix it… probably because the version of svn on my machine is much older than the version of git.

Maybe the notice should just point to the source of the problem and suggest people update their software stacks? That would at least help explain to people why it’s happening?

[Roland Haas]

Yes, I will try and add such a notice. Something along the lines:

Your download may fail if the version of OpenSSL used by subversion and git is too old (older than version 1.0.1 from 2012). In this case you should update your software stack.

I do not really want to provide explicit instructions on how to modify the parfile inline there because they get too long very easily.

Out of curiosity: are you using a Mac (we had issues in the past and needed to require that people install svn from HomeBrew or MacPorts and not use the Apple provided one)? What OS version? Do you use the subversion client that Apple provides or one out of MacPorts/HomeBrew/SomeOtherPackageManager? You should be able to tell be running which svn.

[Jonah Miller]

Ah. Indeed, I’m using a mac. OSX 10.14.6 mojave. And I’m using the system svn out of /usr/bin/svn. Presumably the correct fix is indeed to install it via a real package manager.

I think that message is good. It’s clear enough to a mac user with a package manager what to do in that case, I think.

[Roland Haas]

Mojave is hardly old so I (and I assume you as well) would have expected this to work. In fact I just tried this (on my OSX Mojave 10.14.6 VM that I keep around for this type of stuff):

/usr/bin/svn checkout https://github.com/EinsteinToolkit/ExternalLibraries-HDF5.git/trunk

which worked fine and checkout out the HDF5 library into a directory trunk.

My svn self-reports as:

svn --version
svn: warning: cannot set LC_CTYPE locale
svn: warning: environment variable LANG is en_US.utf8
svn: warning: please check that your locale name is correct
svn, version 1.10.4 (r1850624)
   compiled Aug 26 2019, 15:04:15 on x86_64-apple-darwin18.0.0

Copyright (C) 2019 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.9 (compiled with 1.3.9)
  - handles 'http' scheme
  - handles 'https' scheme

The following authentication credential caches are available:

* Plaintext cache in /Users/jenkins/.subversion
* GPG-Agent
* Mac OS X Keychain

and my env contains only (I logged in remotely):

TERM=screen
SHELL=/bin/bash
TMPDIR=/var/folders/5y/pw0d549j4ggd7r0s7ndkckr00000gn/T/
SSH_CLIENT=10.0.2.2 50054 22
SSH_TTY=/dev/ttys002
USER=jenkins
MAIL=/var/mail/jenkins
PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
LC_COLLATE=C
PWD=/Users/jenkins
LANG=en_US.utf8
SHLVL=1
HOME=/Users/jenkins
LOGNAME=jenkins
SSH_CONNECTION=10.0.2.2 50054 10.0.2.15 22
LC_TIME=en_DK.UTF-8
_=/usr/bin/env

Do you have anything set in your .subversion file or in in environment variable that could affect this? Notably, what is the output you get trying to run the svn checkout command above?

[Jonah Miller]

My checkout returned

svn: E170013: Unable to connect to a repository at URL 'https://github.com/EinsteinToolkit/ExternalLibraries-HDF5.git/trunk'
svn: E000061: Error running context: Connection refused

my svn version is

svn, version 1.9.4 (r1740329)
   compiled Nov  6 2017, 18:06:35 on x86_64-apple-darwin16.1.0

Copyright (C) 2016 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network protocol.
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.8 (compiled with 1.3.8)
  - handles 'http' scheme
  - handles 'https' scheme

The following authentication credential caches are available:

* Plaintext cache in /Users/jonahm/.subversion
* GPG-Agent
* Mac OS X Keychain

I have some proxies enabled in my environment (and in the .subversion/servers file, because of my institutional firewall. That might be interfering somehow. Or the firewall might be the problem.

[Roland Haas]

Ok, I am leaning towards chalking this up to your institution trying to hard to protect you or github refusing to talk to a proxy. The connection refused error is definitely different from what an incompatible OpenSSL library produces (that one says "SSL handshake failed: SSL error: tlsv1 alert protocol version"). I am closing this for good.

[Jonah Miller]

Yeah, that seems reasonable to me.