- removed comment
OSX does not install trusted certificates, breaking GetComponents (svn)
I just tried to check out the ET Hilbert release on my OSX laptop (OSX 10.10.4, subversion version 1.7.19) and get for the LSUThorns: --8<-- Could not checkout module LSUThorns/SummationByParts svn: E175002: Unable to connect to a repository at URL 'https://svn.cct.lsu.edu/repos/numrel/LSUThorns/SummationByParts/branches/ET_2015_05' svn: E175002: OPTIONS of 'https://svn.cct.lsu.edu/repos/numrel/LSUThorns/SummationByParts/branches/ET_2015_05': Server certificate verification failed: issuer is not trusted (https://svn.cct.lsu.edu) --8<-- https://www.sslshopper.com/ssl-checker.html#hostname=https://svn.cct.lsu.edu reports the certificate to be ok though (with the exception of it using SHA1 as a hash).
Doing a manual svn checkout https://svn.cct.lsu.edu/repos/numrel/LSUThorns/QuasiLocalMeasures/branches/ET_2015_05 asks me whether I would want to trust the certificate.
- I thought the LSU certificates were trusted by common OS by default by now (and OSX is certainly common)
- I thought we had special code in GetComponents to make it automatically not try and verify signatures because of this
-
this is really becoming a nuisance (if indeed caused by an uncommon certificate issuer and not by something odd on my laptop) :-)
openssl ssl_client -connect svn.cct.lsu.edu:443 </dev/null >ssl.out reports a self-signed certficate though this may well be the untrusted certificate.
Keyword:
Comments (41)
-
reporter -
- removed comment
I think we should move the LSUThorns thorns to Bitbucket: - LSUThorns/PeriodicCarpet - LSUThorns/QuasiLocalMeasures - LSUThorns/SummationByParts - LSUThorns/Vectors
-
reporter - removed comment
Suggestions:
- PeriodicCarpet into Carpet
- QuasiLocalMeasures into EinsteinAnalysis
- SummationByParts into CactusNumerical (needs license to be LGPL) or Numerical if code does not adhere to the stricter Cactus rules (ie it really needs documenation for that)
- Vectors into CactusUtils or Utils (same as SummationByParts)
-
- removed comment
Thorns moving sounds fine, but this is generally not the right place to discuss it. More people might be interested that are not watching this issue.
Concerning the issue: can this be closed as 'worksforme', since this seems to apply only to one user? Maybe this helps? http://jw35.blogspot.gr/2011/02/root-certificates-for-macos-openssl.html
-
reporter - removed comment
The suggestion of setting up a cert.pem in /System/Library/OpenSSL seems to no longer work. OpenSSL still complains and running it through dtruss (strace equivalent) reveals it is not even trying to open any cert.pem file.
-
- removed comment
Maybe this - or something else in the zoo of pages about this problem on OSX might help. I can't help but think that distributing openssl without a pointer to the root certificates is kind of pointless, and breaks things in unexpected ways. If this is really the problem, not even an older certificate would help.
http://movingpackets.net/2015/03/18/telling-openssl-about-your-root-certificates/
-
- removed comment
This is also a problem on my Raspberry PI (with all packages up to date).
-
- removed comment
Replying to [comment:7 eschnett]:
This is also a problem on my Raspberry PI (with all packages up to date).
I can confirm that, using an up2date Raspbian. This is based on the old Debian wheezy, which points to the same problem: the ca-certificates are from Jan 2013. In theory this should be enough, since the root certificate was issued in 2010, but apparently it wasn't included in Raspbian until after Jan 2013. I would expect that to go away once Raspbian uses the Jessie release.
-
- removed comment
I propose to close this ticket. The "offending" root cert is by now 5 years old. It shouldn't take that long to propagate to clients - 5 years is already a long time for a certificate, and 1/4 of its lifetime. Imho, this is an issue with old clients. We cannot do without certificates for write access, and using certificates requires to have your trusted certs somewhat updated.
-
reporter - removed comment
I would think that even if the offending client is old (and I think we all believe that it is) we still need to support it if the old client is very prevalent. The raspberry is probably not something that is common enough, however OSX would be I think. However for OSX since its OpenSSL seems to lack any root certificates it would seem to me that all attempts to use https transport with openssl (and thus svn) must fail. Is this correct?
-
- removed comment
The issue with certificates has been around for years, has not been solved yet, and continues to annoy people. What you probably mean (Frank) is that there is no way to solve this with our current repo infrastructure (hosting SVN repos at LSU). That doesn't mean that there isn't another way, or that the issue isn't annoying to people any more.
We should address this; if we can't get certificates to work, and if the "solution" is to ignore them, then we should use a different mechanism and forego svn via https. For example plain http should work, or svn+ssh, or switching to git.
-
- removed comment
Plain http wouldn't work for write access. We could use it for read-only, but then have the problem with people trying to use it to commit again. svn+ssh wouldn't work in this case (no ssh access), but also kind of circumvents the problem, as there is no trusted host-key. This would be similar to telling svn to trust all certificates (bad idea, but also a possibility - could be done for problematic certs only too). git isn't solving this at all. git also uses either ssh or https as transport protocol - both with the same issues as already discussed. This isn't a svn problem.
The root problem is clients not updating trusted certs, and this isn't something we could fix as ET developers. I trust that all OSX users are loudly complaining to Apple? In this case (afaik), no certificate would work. Isn't there a version of subversion/openssl available for OSX that isn't broken (not shipping with trusted root certs counts for me as "broken")? We could then require (and test for) this.
-
- removed comment
Bitbucket and Github solved the problem. We can either do something similar, or switch to them (or another service provider).
-
- removed comment
How did they solve this problem? I can only imagine that the git client shipped with OSX is not as broken as the svn client, in which case bitbucket/github didn't solve anything. What does the OSX client say to
svn co https://github.com/ianhinder/Kranc
Really - if Apple svn/openssl is broken on OSX, our reaction shouldn't be to abandon svn or https. What do we switch to if some OS breaks git next? We have to get svn working instead. Either users complain until it is fixed by the vendor (you do pay, don't you?), or they use a working version of the client.
So, again my question: Is there a working version of subversion/openssl available for OSX? I don't have such a machine and cannot try.
-
- removed comment
[Long answer deleted.]
I don't care about the details, but I expect downloading the code to work out of the box. I don't think we should close a ticket when there is no good resolution.
-
- removed comment
I also expect the svn client to work out of the box. Apparently it doesn't. Is there working version available via a standard mechanism - homebrew maybe? Let's not close this ticket then, but let's find a solution on OSX. But for that I need someone to test and try. There seem to be plenty available: https://subversion.apache.org/packages.html#osx . If one of these works and is reasonably easy to install, all we need to do is blacklist the broken version.
-
reporter - removed comment
Replying to [comment:14 knarf]:
How did they solve this problem? I can only imagine that the git client shipped with OSX is not as broken as the svn client, in which case bitbucket/github didn't solve anything. What does the OSX client say to
svn co https://github.com/ianhinder/Kranc
It says:queech4u:tmp rhaas$ /usr/bin/svn co https://github.com/ianhinder/Kranc kranc Error validating server certificate for 'https://github.com:443': - The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually! Certificate information: - Hostname: github.com - Valid: from Apr 8 00:00:00 2014 GMT until Apr 12 12:00:00 2016 GMT - Issuer: www.digicert.com, DigiCert Inc, (null), (null), US ((null)) - Fingerprint: A0:C4:A7:46:00:ED:A7:2D:C0:BE:CB:9A:8C:B6:07:CA:58:EE:74:5E (R)eject, accept (t)emporarily or accept (p)ermanently? ^Csvn: E200015: Unable to connect to a repository at URL 'https://github.com/ianhinder/Kranc'
so svn also fails for bitbucket's svn transport (the svn that works for me is the homebrew one '''after''' I manually install a cert.pem file eg from curl).
I think that for downloads we can live with http transports and the number of persons who commit is small enough (and expert enough) that we can ask those to use a fully fledged svn and openssl client.
I also think that we must make every effort so that downloads work. Basically no matter how annoying this is for us.
-
- removed comment
Ok, we could let GetComponents switch https to http if this client is detected. We might be able to see this from 'svn --version', but really svn isn't to blame here, but an incomplete openssl installation. 'openssl version' might give us a clue, but again - we really would have to check if svn uses an openssl without certificates. This seems to be quite complicated. Maybe we can simply blacklist the default OSX svn client (regardless of openssl configuration), and in that case switch to http?
-
reporter - removed comment
One more data point: I can get the default client to work by setting the
SSL_CERT_FILE
environment variable to the certificate bundle from curl. Unfortunately that file does not exist on a freshly installed mac it seems. However one can extract the keys out of OSX's key chain using:security export -k /System/Library/Keychains/SystemRootCertificates.keychain -t certs -f pemseq p
so we '''could''' (when we detect we are running on OSX and that certificates are failing, eg by checking that
svn info --non-interactive
fails butsvn info --non-interactive --trust-server-cert
succeeds so that we don't have to rely on some English language string parsing) use the "security" command to write the certificates into a file (either in /tmp or in $HOME/.crl) and then setSSL_CERT_FILE
before we call svn. -
- removed comment
You cannot compile the ET on a freshly installed OS X system without a lot of extra packages, usually provided by MacPorts or Homebrew. The subversion (/usr/bin/svn) that you are trying to use is installed by xcode (which is needed for compiling on OS X), and for some reason seems to be broken regarding SSL. We could blacklist this version and insist that users install a good subversion. The version in MacPorts has always worked for me. I have just tested the MacPorts and Homebrew versions on github, and they both work, whereas the xcode version does not. Since we only support using OS X with MacPorts or Homebrew (and people compiling everything from scratch are well-capable of building their own SVN), I suggest we just tell people to install svn from whichever package manager (macports or homebrew) they are using. This would be done when GetComponents detects that the svn version is the apple version.
I would rather not add complicated fallback logic into GetComponents to try to work with buggy software, when it is a small extra step for users to add one more package to the list of recommended packages that they need to install from their package manager.
-
reporter - removed comment
If this failure happened anywhere other than the initial download I would agree that we can ask users to update their system. However since this is the very first thing a new user will do (in particular following the Download link on our website) this really should work, no matter what. Even with the old svn client (I think). If the solution of writing a "useful" set of certificates using OSX's built-in certificate store is not acceptable, then I think that at the very least we must add code to GetComponents that detects that svn is failing due to missing certificates and if it also finds that it runs on OSX it must output a warning explaining what is happening and how to fix it (by suggesting to install homebrew or macports). This warning must be collected with all the other warnings and output at the end of the GetComponents output. I really believe that if the initial download does not work, a new potential user will just give up and look somewhere else (unless they are in a group already using the ET in which case they have local support to sort this out).
-
- changed title to OSX does not install trusted certificates, breaking GetComponents (svn)
- marked as
- changed component to GetComponents
- removed comment
-
- removed comment
This is not just an OSX problem.
I witnessed the same error message on RHEL6 (Stampede), preventing me from compiling. In the end, I just grabbed the tarball and added the LSUThorns package manually. Given how widespread RHEL6 is, increased priority is justified.
I do not see this behavior on Ubuntu 14.04.3.
-
- removed comment
This is a separate issue from the OSX problems. I opened a separate ticket for it:
#1812. -
reporter - changed status to open
- removed comment
See pull request: https://github.com/gridaphobe/CRL/pull/1
-
- removed comment
This seems to show the message if on OSX and certificate validation fails. Isn't that a little broad? It could fail for a valid reason.
-
reporter - removed comment
True. Right now it shows the message about the stock svn client if svn info first fails (line 584) and then succeeds if one adds --trust-server-cert (line 587). The man page says "--tryst-server-cert}} that this tell svn to "accept SSL server certificates from unknown certificate authorities" so other causes of failure to verify (expired, faulty) should still trigger. If the failure is due to something other than the certificates then this would have to have changed in between the two svn calls for this check to become invalid. It is of course possible that the validation fails for another certificate related reason and
--trust-server-cert
accepts this. On could make the regex more explicit since svn actually outputs:Server certificate verification failed: issuer is not trusted
. Would you think this to be better, it would at least avoid mis-classifying expired certificates?I am a little bit unhappy to look for an English language text to determine some success state but the text seems to stay the same even if I eg set LANG to "DE" to try and see if I can get German language error messages. A bigger concern may be subversion updating the error text in the future.
-
- removed comment
It turns out that we probably need a similar fix also for clients that happen to not have updated root certs. Thus, I propose to make this change a little more general: in addition to disabling cert-checking and/or explicitly specifying the ca cert when this fails on OSX, do this on all systems where this fails - if the problem occurs with servers we know can fail. We can either hardcode these servers in GetComponents, or add something to the thornlist.
-
reporter - removed comment
Has there been any progress an a more comprehensive fix? If not I think we should include the warning and workaround in the release. I have updated the pull request to test not only on OSX but on all OS.
-
reporter - removed comment
Anything? Given that we are about to release I assume this is now working on OSX? Conceivably using subversion from macports/homebrew? Should we include a warning to this effect in the release notes?
-
- removed comment
If clients are incorrectly installed, we have not really an option there. Disabling certs if their checking fails amounts to disabling certificates. At this point, I would opt for mentioning this in the release notes, however, I would need some current information about this from actual users.
-
- removed comment
It still fails with the OS X system version of svn, but it works fine with the homebrew version installed.
-
- removed comment
Thanks. I'll mention that Apple screwed this up, and you need homebrew for a working svn client.
-
reporter - removed comment
Just to repeat what I said 8 months ago (https://trac.einsteintoolkit.org/ticket/1801#comment:29) : --8<-- Has there been any progress an a more comprehensive fix? If not I think we should include the warning and workaround in the release. I have updated the pull request to test not only on OSX but on all OS. --8<--
Given that there is no more comprehensive solution yet I would apply this patch which will at least give users the option of manually accepting all those certificates.
-
- removed comment
It would be a step in the right direction. It wouldn't cover all cases (would be OSX only), but that's probably ok.
Out of curiosity: did any of the OSX users file a bug report with Apple - since this is quite obviously a bug on their system?
-
reporter - removed comment
Does this mean "please apply"?
Note that this is no longer specific to OSX --8<-- I have updated the pull request to test not only on OSX but on all OS. --8<-- so all OS display the warning (the one for OSX is just more verbose) and all OS will then allow users to manually accept the unknown certificates.
I have not filed a bug nor would I expect this to help I lot I admit. I do not even know how to file bugs for OSX.
-
- removed comment
Yes, this is a "please apply". Making the user accept also takes care of the security implications this otherwise might have.
As for Apple: I suspect this could be it: https://developer.apple.com/bug-reporting/
-
reporter - changed status to open
- removed comment
Thank you. I set the "reviewed ok" state.
-
reporter - removed comment
Hah, seems as if Apple fixed this issue in at least the latest OSX release. Even after wiping $HOME/.subversion (and uninstalling subversion from homebrew) and on a fairly newly installed OSX laptop I do not get any failures for
/usr/bin/svn info http://svn.cactuscode.org/projects/ExternalLibraries/hwloc/
This also seems to happen to others. I do not know however in which OSX release this was fixed.
Question now: do we want to keep the workaround with the always present danger of it introducing new unknown bugs (rather than old known ones)?
The test still triggers though in case things are actually wrong, eg if one uses ssh to forward port 443 from ones laptop to svn.cactuscode.org then
/usr/bin/svn info http://localhost/projects/ExternalLibraries/hwloc/
fails due to a mismatch in the certificates (cn-mismatch).
-
reporter - changed status to resolved
- removed comment
Applied as git hash b40c9bb of CRL.
-
reporter - edited description
- changed status to closed
- Log in to comment
Well this seems to be something either local to my mac or at least local to OSX. I can check out from my Linux box and the svn client also complains about the aei svn server (which tends to not happen normally).