Thornlists entry for external libraries on github is broken

Create issue
Issue #2293 wontfix
Jonah Miller created an issue

The entry for external libraries in the thornlist looks like this:

# Additional Cactus thorns
!TARGET   = $ARR
!TYPE     = svn
!URL      = https://github.com/EinsteinToolkit/$1-$2.git/trunk

this is a git repo, not an svn repo. So the entry should be

# Additional Cactus thorns
!TARGET   = $ARR
!TYPE     = git
!URL      = https://github.com/EinsteinToolkit/$1-$2.git

I’m happy to submit a PR. But I wanted to report the bug first, since I might not get to it today. This appears to be present in both the development version and Proca.

Comments (11)

  1. Roland Haas

    This is intentional (and should be working, do you observe a failure while checking out the code?). GitHub supports accessing the (git) repositories via subversion. In the ET we use this to only download the (large) tarballs for the current release of the ExternalLibrary and not all tarballs stored in history. See https://help.github.com/en/articles/support-for-subversion-clients for GitHub’s doc. I cannot find a ticket about the change right now, but it was discussed in some of the ET calls so should be in the minutes (somewhere…).

    If you stumbled across this because GitHub refused to talk to your cluster’s subversion client, then the reason for the failure is most likely a too old OpenSSL library on your cluster.

  2. Jonah Miller reporter

    Ah, I see! Yes, I discovered this when github refused to talk to my machine. Okay, I’ll mark this as resolved.

    It might be worth mentioning on the downloads page that the user needs an up-to-date version of OpenSSL. And/or provide an alternative version of the thornlist for those with out-of-date libraries.

  3. Roland Haas

    The notice can be added but I do not know of a workaround that I could provide (other than: download on your laptop then use simfactory/bin/sim sync to copy to the cluster). Since the OpenSSL library itself is to blame, switching from svn to git does not fix the issue, unless of course your git installation (eg via an environment module) links to a newer version of openssl than your svn executable (which may be part of the OS packages). The latter is the case on BW by default.

    I tried switching https:// to http:// but github just redirects that back to https://.
    GitHub’s announcement (https://github.blog/2018-02-01-crypto-removal-notice/) does not say but apparently tlsv1.2 support is in OpenSSL since 2012 (https://stackoverflow.com/questions/48178052/when-was-tls-1-2-support-added-to-openssl) which is unfortunately just “recent” enough that certain clusters with frozen software stacks (such as Blue Waters) are affected.

  4. Jonah Miller reporter

    The failure I encountered was actually on my Apple laptop, and switching to git did fix it… probably because the version of svn on my machine is much older than the version of git.

    Maybe the notice should just point to the source of the problem and suggest people update their software stacks? That would at least help explain to people why it’s happening?

  5. Roland Haas

    Yes, I will try and add such a notice. Something along the lines:

    Your download may fail if the version of OpenSSL used by subversion and git is too old (older than version 1.0.1 from 2012). In this case you should update your software stack.

    I do not really want to provide explicit instructions on how to modify the parfile inline there because they get too long very easily.

    Out of curiosity: are you using a Mac (we had issues in the past and needed to require that people install svn from HomeBrew or MacPorts and not use the Apple provided one)? What OS version? Do you use the subversion client that Apple provides or one out of MacPorts/HomeBrew/SomeOtherPackageManager? You should be able to tell be running which svn.

  6. Jonah Miller reporter

    Ah. Indeed, I’m using a mac. OSX 10.14.6 mojave. And I’m using the system svn out of /usr/bin/svn. Presumably the correct fix is indeed to install it via a real package manager.

    I think that message is good. It’s clear enough to a mac user with a package manager what to do in that case, I think.

  7. Roland Haas

    Mojave is hardly old so I (and I assume you as well) would have expected this to work. In fact I just tried this (on my OSX Mojave 10.14.6 VM that I keep around for this type of stuff):

    /usr/bin/svn checkout https://github.com/EinsteinToolkit/ExternalLibraries-HDF5.git/trunk
    

    which worked fine and checkout out the HDF5 library into a directory trunk.

    My svn self-reports as:

    svn --version
    svn: warning: cannot set LC_CTYPE locale
    svn: warning: environment variable LANG is en_US.utf8
    svn: warning: please check that your locale name is correct
    svn, version 1.10.4 (r1850624)
       compiled Aug 26 2019, 15:04:15 on x86_64-apple-darwin18.0.0
    
    Copyright (C) 2019 The Apache Software Foundation.
    This software consists of contributions made by many people;
    see the NOTICE file for more information.
    Subversion is open source software, see http://subversion.apache.org/
    
    The following repository access (RA) modules are available:
    
    * ra_svn : Module for accessing a repository using the svn network protocol.
      - with Cyrus SASL authentication
      - handles 'svn' scheme
    * ra_local : Module for accessing a repository on local disk.
      - handles 'file' scheme
    * ra_serf : Module for accessing a repository via WebDAV protocol using serf.
      - using serf 1.3.9 (compiled with 1.3.9)
      - handles 'http' scheme
      - handles 'https' scheme
    
    The following authentication credential caches are available:
    
    * Plaintext cache in /Users/jenkins/.subversion
    * GPG-Agent
    * Mac OS X Keychain
    

    and my env contains only (I logged in remotely):

    TERM=screen
    SHELL=/bin/bash
    TMPDIR=/var/folders/5y/pw0d549j4ggd7r0s7ndkckr00000gn/T/
    SSH_CLIENT=10.0.2.2 50054 22
    SSH_TTY=/dev/ttys002
    USER=jenkins
    MAIL=/var/mail/jenkins
    PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
    LC_COLLATE=C
    PWD=/Users/jenkins
    LANG=en_US.utf8
    SHLVL=1
    HOME=/Users/jenkins
    LOGNAME=jenkins
    SSH_CONNECTION=10.0.2.2 50054 10.0.2.15 22
    LC_TIME=en_DK.UTF-8
    _=/usr/bin/env
    

    Do you have anything set in your .subversion file or in in environment variable that could affect this? Notably, what is the output you get trying to run the svn checkout command above?

  8. Jonah Miller reporter

    My checkout returned

    svn: E170013: Unable to connect to a repository at URL 'https://github.com/EinsteinToolkit/ExternalLibraries-HDF5.git/trunk'
    svn: E000061: Error running context: Connection refused
    

    my svn version is

    svn, version 1.9.4 (r1740329)
       compiled Nov  6 2017, 18:06:35 on x86_64-apple-darwin16.1.0
    
    Copyright (C) 2016 The Apache Software Foundation.
    This software consists of contributions made by many people;
    see the NOTICE file for more information.
    Subversion is open source software, see http://subversion.apache.org/
    
    The following repository access (RA) modules are available:
    
    * ra_svn : Module for accessing a repository using the svn network protocol.
      - handles 'svn' scheme
    * ra_local : Module for accessing a repository on local disk.
      - handles 'file' scheme
    * ra_serf : Module for accessing a repository via WebDAV protocol using serf.
      - using serf 1.3.8 (compiled with 1.3.8)
      - handles 'http' scheme
      - handles 'https' scheme
    
    The following authentication credential caches are available:
    
    * Plaintext cache in /Users/jonahm/.subversion
    * GPG-Agent
    * Mac OS X Keychain
    

    I have some proxies enabled in my environment (and in the .subversion/servers file, because of my institutional firewall. That might be interfering somehow. Or the firewall might be the problem.

  9. Roland Haas

    Ok, I am leaning towards chalking this up to your institution trying to hard to protect you or github refusing to talk to a proxy. The connection refused error is definitely different from what an incompatible OpenSSL library produces (that one says "SSL handshake failed: SSL error: tlsv1 alert protocol version"). I am closing this for good.

  10. Log in to comment