New TLS certificate for einsteintoolkit.org is self-signed and gives browser warnings/errors

[Ian Hinder]

See https://www.sslshopper.com/ssl-checker.html?hostname=einsteintoolkit.org.

einsteintoolkit.org resolves to 130.39.21.27

Server Type: Apache/2.4.25 (Debian)

The certificate will expire in 364 days.

The hostname (einsteintoolkit.org) is correctly listed in the certificate.

The certificate is self-signed. Users will receive a warning when accessing this site unless the certificate is manually added as a trusted certificate to their web browser. You can fix this error by buying a trusted SSL certificate
    Common name: einsteintoolkit.org
Organization: Louisiana State University Org. Unit: Center for Computation and Technology
Location: Baton Rouge, Louisiana, US
Valid from April 24, 2019 to April 23, 2020
Serial Number: 10477637556223891899 (0x91680bd8300459bb)
Signature Algorithm: sha256WithRSAEncryption
Issuer: einsteintoolkit.org

Safari throws up a big warning and doesn't let you visit the site unless you confirm a security exception with your login password.

Didn't we have an automated tool to check these things and report them?

[Steven R. Brandt]

We had a failure of the server where the machines were hosted, and the docker images were restarted on different servers but the certificates were lost. We are in the process of getting new ones.

[Roland Haas]

There is no automated tool. The closest we have is Frank's monitor site: https://www.cct.lsu.edu/~knarf/cgi-bin/monitor.cgi uptimerobot in the free version does not check ssl certs :-(

[Steven R. Brandt]

I have a tool which monitors websites and certs (it's a 50 line python script). If anyone is interested, I can share it.

[Ian Hinder]

Yes please! I have been interested recently in monitoring solutions, and have played with Icinga (a nagios fork). Unfortunately, its certificate checking is restricted to checking expiry dates, not certificate chains. Maybe your tool could be hooked up to it instead? Would it have caught this problem? I found an SSL/TLS checker (https://github.com/drwetter/testssl.sh), but it is very full-featured, takes a long time to run (maybe minutes), and doesn't return a simple exit code for "good" or "bad", probably because most sites will fail at least some of its checks (e.g. using old ciphers). Probably a simple grep on the output would catch the things we need, and maybe taking a few minutes doesn't matter if we only run it infrequently, but if your solution is much simpler, then I like it!

Was there no backup of the Einstein Toolkit server, in which the certificates could be found?

[Ian Hinder]

For reference, the following docker command will run a test on einsteintoolkit.org and catch the current problem. It takes 26 s to run, which is better than the 3 min taken by the full suite of tests.

mkdir logs
time sudo docker run -ti -v $PWD/logs:/tmp/logs drwetter/testssl.sh --server-defaults --color 0 --quiet --severity HIGH -oC /tmp/logs -oJ /tmp/logs -oL /tmp/logs -oH /tmp/logs einsteintoolkit.org

The standard output is human readable:

 Start 2019-04-26 12:36:50        -->> 130.39.21.27:443 (einsteintoolkit.org) <<--

 rDNS (130.39.21.27):    einsteintoolkit.org.
 Service detected:       HTTP


 Testing server defaults (Server Hello) 

 TLS extensions (standard)    "server name/#0" "renegotiation info/#65281"
                              "EC point formats/#11" "session ticket/#35"
                              "heartbeat/#15"
                              "application layer protocol negotiation/#16"
 Session Ticket RFC 5077 hint 300 seconds, session tickets keys seems to be rotated < daily
 SSL Session ID support       yes
 Session Resumption           Tickets: yes, ID: yes
 TLS clock skew               Random values, no fingerprinting possible 
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits
 Server key usage             --
 Server extended key usage    --
 Serial / Fingerprints        91680BD8300459BB / SHA1 A06674AE0582BD65DFA9E5A43F28F32B59A29FF4
                              SHA256 80412717A02D2DE5FA41D6D26DE378397278F2983BE33AFEA9C6E6B75F1E8FCE
 Common Name (CN)             einsteintoolkit.org
 subjectAltName (SAN)         missing (NOT ok) -- Browsers are complaining
 Issuer                       self-signed (NOT ok)
 Trust (hostname)             via CN only -- Browsers are complaining (same w/o SNI)
 Chain of trust               NOT ok (self signed)
 EV cert (experimental)       no 
 "eTLS" (visibility info)     not present
 Certificate Validity (UTC)   363 >= 60 days (2019-04-24 21:33 --> 2020-04-23 21:33)
 # of certificates provided   1
 Certificate Revocation List  --
 OCSP URI                     --
                              NOT ok -- neither CRL nor OCSP URI provided
 OCSP stapling                not offered
 OCSP must staple extension   --
 DNS CAA RR (experimental)    not offered
 Certificate Transparency     --


 Done 2019-04-26 12:37:14 [  25s] -->> 130.39.21.27:443 (einsteintoolkit.org) <<--

And the CSV file can be parsed by awk to give:

$ awk -vFPAT='([^,]*)|("[^"]+")' -vOFS="\t" '{print $1,$2,$4,$5}' logs/einsteintoolkit.org_p443-*.csv | column -t -s $'\t'

"id"                   "fqdn/ip"                           "severity"  "finding"
"service"              "einsteintoolkit.org/130.39.21.27"  "INFO"      "HTTP"
"cert_subjectAltName"  "einsteintoolkit.org/130.39.21.27"  "HIGH"      "No SAN, browsers are complaining"
"cert_caIssuers"       "einsteintoolkit.org/130.39.21.27"  "CRITICAL"  "selfsigned"
"cert_trust"           "einsteintoolkit.org/130.39.21.27"  "HIGH"      "via CN only (same w/o SNI)"
"cert_chain_of_trust"  "einsteintoolkit.org/130.39.21.27"  "CRITICAL"  "failed (self signed)."
"cert_revocation"      "einsteintoolkit.org/130.39.21.27"  "HIGH"      "Neither CRL nor OCSP URI provided"

[Roland Haas]

If all we want is a simple checker then Frank's code (behind) https://www.cct.lsu.edu/~knarf/cgi-bin/monitor.cgi will do as well as the python code that Steve mentioned. Note that this would still need to run somewhere so if that host running it goes down we are no better off than now. Note that for the current instance of self-signed certificates this was not something that happened without people being aware. As far as I know Steve was aware of the certificates being incorrect from the beginning but given that the servers were brought back online at 9pm local time, getting back the original certs was delayed.

There was unfortunately no explicit email to the mailing list (once it was back up) stating "we are sorry, einsteintoolkit.org and docs.einsteintoolkit.org are down" and "einsteintoolkit.org is back but you will have to accept the temporary certificate".

Speaking of which: having everything back up and running before Tuesday next week would be very nice.

I am not sure if one can restrict what to watch to particular components in bb but I can change my filter to let "website" components through and only throw out the others (I need the filter b/c bitbucket makes me auto-watch all tickets to which I have responded).

[Steven R. Brandt]

fixed

[Ian Hinder]

According to SSL Checker, the required intermediate certificates have not been installed: https://www.sslshopper.com/ssl-checker.html?hostname=einsteintoolkit.org.

The certificate is not trusted in all web browsers. You may need to install an Intermediate/chain certificate to link it to a trusted root certificate. Learn more about this error. The fastest way to fix this problem is to contact your SSL provider.

[Roland Haas]

As a side note: this certificate chain missing is not caught by: https://www.cct.lsu.edu/~knarf/cgi-bin/monitor.cgi (and the “not all” is correct, my browser is happy with the website right now).

The raw

openssl s_client -showcerts -connect einsteintoolkit.org:443 </dev/nul

does report “Verify return code: 21 (unable to verify the first certificate)” though.

[Steven R. Brandt]

intermediate and bundle installed.

[Ian Hinder]

Thanks! The direct parent certificate, AddTrust External CA Root, seems to be using a SHA1 cert, known to be weak since 2012, as reported by ssl-checker. I suppose this is probably beyond our control, but it’s a bit disappointing not to get “full marks”! Apparently there are lots of questions and confusion on the net about intermediate certs being flagged, and how in some cases, it’s ok for some of the certs (maybe only the root?) to use SHA1. Safari, at least, is perfectly happy with it.

One of the certificates is signed with a SHA1 signature. We recommend that you reissue or replace this certificate with one that uses a SHA-2 signature. Contact your SSL provider about how to do this. Read more about the SHA-1 deprecation here.