Error authenticating with bbcloud

[Shaun Martinec]

I was able to authenticate to our hosted Jira, but have been unable to authenticate to our Bitbucket cloud account. Getting these messages with debug on:

[2020-03-27 13:52:08:015] Atlassian for VS Code (v2.4.10) activated in 17 ms
[2020-03-27 13:52:08:145] Online check attempt 1 failed. There are 4 retries left.
[2020-03-27 13:52:08:148]  Error: Error getting pipes Error: certificate has expired
[2020-03-27 13:52:08:148] knownpipes 
[2020-03-27 13:52:09:149] Online check attempting to connect to http://atlassian.com
[2020-03-27 13:52:09:149] Online check attempting to connect to https://bitbucket.org
...
[2020-03-27 13:52:23:481] Online check attempt 5 failed. There are 0 retries left.
[2020-03-27 13:52:23:481] You have gone offline :(
[2020-03-27 13:54:44:587] auth server started on port 31415
[2020-03-27 13:54:47:043]  Error: Error fetching Bitbucket tokens: Error: certificate has expired
[2020-03-27 13:54:47:046] Error authenticating Error authenticating with bbcloud: Error: Error fetching Bitbucket tokens: Error: certificate has expired

And here is the error in the browser:

MacOS Catalina: 10.15.3

Google Chrome: 80.0.3987.149

[Jonathan Doklovic]

@Shaun Martinec That’s a strange one, and I can’t re-create it. Can you try again and see if it was a temporary blip? Also, do you use a custom certificate and/or proxy with your self-hosted Jira?

[Shaun Martinec]

I’ve been trying for days, so it’s not a temporary issues. We do use a certificate for our self-hosted Jira in order to support SSL, but it’s GlobalSign Root CA with AlphaSSL CA intermediate. So, it should be supported in any browser/client. Also, how is that related since it’s the Bitbucket side that’s failing and Jira is working fine in VScode?

[Shaun Martinec]

@{557057:3695b794-cde3-43a2-99aa-c0fc6151b754} Any ideas?

[Jonathan Doklovic]

Just trying to figure out what’s causing the error… I’m wondering if bucket requests are being forced through a proxy or during and then hitting your SSL setup and failing somehow.

Do you have proxy environment variables setup?

[Shaun Martinec]

No proxy. I work from home on FIOS and plugged directly into router. I can access bitbucket.org fine.

[Jonathan Doklovic]

@Shaun Martinec From your error log, it looks like the online check is failing to connect to bitbucket.org. This is just a simple HEAD request to the site to check for connectivity.

Seems like there may be something getting in the way of these requests, maybe a firewall?

If you open up “Open Settings (UI)” and then search for “curl” you should see an option to “Enable curl logging”. This will output every rest call our extension makes as a curl command. Once you do that, you can grab the curl command it’s making to bitbucket.org and run it in a terminal to see what happens. In the terminal you can also pass -v to get more verbose output.

[Shaun Martinec]

@{557057:3695b794-cde3-43a2-99aa-c0fc6151b754}

Here is command from output:

MACBOOK2:~ shaun$ curl -X HEAD "https://bitbucket.org"
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
Warning: way you want. Consider using -I/--head instead.
MACBOOK2:~ shaun$ 

Here is result suggested by Warning message:

MACBOOK2:~ shaun$ curl -I "https://bitbucket.org"
HTTP/1.1 200 OK
server: envoy
Cache-Control: max-age=900
content-type: application/octet-stream
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
date: Mon, 30 Mar 2020 22:34:13 GMT
x-envoy-upstream-service-time: 0
Connection: Keep-Alive
X-Cache-Info: cached
content-length: 0

MACBOOK2:~ shaun$ 

Looks like you need to update to the new format. More detail here: https://serverfault.com/questions/140149/difference-between-curl-i-and-curl-x-head

[Jonathan Doklovic]

@Shaun Martinec The curl command is only spit out in the logs for debugging and is not used internally to make requests, so it should be fine. Since the curl command completes successfully on the command line, it seems like it’s something in the client configuration that’s getting in they way.

Also, it looks like you actually did authenticate successfully with Bitbucket and it’s just a problem with the client making rest requests.

Juest for a sanity check, can you please run these in a terminal and make sure they are blank:

echo $HTTPS_PROXY
echo $HTTP_PROXY

Finally can you open your keychain manager and see if there’s a key named “atlascode-authinfoV2/bitbucket ?

If so, you can copy the “password” for it and paste it into a text editor (it’s just json) to see if it has a refresh token.

[Shaun Martinec]

@{557057:3695b794-cde3-43a2-99aa-c0fc6151b754}

No proxy:

MACBOOK2:~ shaun$ echo $HTTPS_PROXY

MACBOOK2:~ shaun$ echo $HTTP_PROXY

MACBOOK2:~ shaun$ 

I do not have a “atlascode-authinfoV2/bitbucket” entry.

I do have a “atlascode-authinfoV3” entry:

{"password":"********","user":{"displayName":"Shaun Martinec","id":"shaun","email":"shaun@******.com","avatarUrl":"https://www.gravatar.com/avatar/43ca3b3707f4e50dda2f36932cd4d073?d=mm&s=48"},"username":"shaun"}

‌

[Jonathan Doklovic]

@Shaun Martinec sorry, atlascode-authinfoV3 is the correct entry. Do you have them for /jira and /bitbucket or just /jira ?

[Shaun Martinec]

I just have the 1 for jira:

‌

[Jonathan Doklovic]

@Shaun Martinec Ok, this is getting a bit “head scratchy”, but there are a few more things we can check…

Can you look in your settings.json and search for these two strings:

enableHttpsTunnel

enableCharles

and let me know if either of those are set to true. (if they are, set them to false and try re-authing)

[Shaun Martinec]

Neither is set.

Why don’t you think it’s related to the “curl -X HEAD” issue?

I was able to confirm that I can retrieve the token manually:

VS Code Atlassian Log:

[2020-03-31 20:43:45:310] auth server started on port 31415
[2020-03-31 20:43:52:141] ----------------------------------------------------------------------
[2020-03-31 20:43:52:141] curl -X POST -H "Content-Type:application/x-www-form-urlencoded" -H "Authorization:Basic ********" --data 'grant_type=authorization_code&code=*******' "https://bitbucket.org/site/oauth2/access_token"
[2020-03-31 20:43:52:141] ----------------------------------------------------------------------
[2020-03-31 20:43:52:190]  Error: Error fetching Bitbucket tokens: Error: certificate has expired
[2020-03-31 20:43:52:193] Error authenticating Error authenticating with bbcloud: Error: Error fetching Bitbucket tokens: Error: certificate has expired

Manual Run in Terminal:

MACBOOK2:~ shaun$ curl -X POST -H "Content-Type:application/x-www-form-urlencoded" -H "Authorization:Basic ********" --data 'grant_type=authorization_code&code=*********' "https://bitbucket.org/site/oauth2/access_to
ken"
{"access_token": "********", "scopes": "pipeline account project pullrequest:write issue:write team snippet:write", "expires_in": 7200, "refresh_token": "********", "token_type": "bearer"}
MACBOOK2:~ shaun$ 

‌

[Jonathan Doklovic]

@Shaun Martinec The curl commands are just logged out for debugging purposes. Inside the extension we don’t actually use curl at all and instead make requests with the axios http client library.

The fact that it runs successfully in your terminal and not within the extension sounds like something is telling the axios client to use a certificate or a proxy or something. It’s difficult to figure out why the http client fails when the terminal does not.

[Shaun Martinec]

Any way to debug axios? If not, can I just manually create the proper Keychain entry?

[Jonathan Doklovic]

@Shaun Martinec You can use the Charles Web Debugger to view/debug all the requests that are made by the axios client in our extension. It’s a rather tedious setup process but you only need to do it once.

There are complete instructions in this comment:
https://bitbucket.org/atlassianlabs/atlascode/issues/186/it-looks-like-youve-gone-offline#comment-54173068

If we don’t get any new info from that, I might be able to add specific debugging in the extension itself and give you a custom build to help us through this, but let’s hope we don’t need to do that

[Shaun Martinec]

This is even crazier - works fine when Charles is enabled. Stops working and I get the certificate error again as soon as I disable Charles. So, even though I have a valid entry in the keychain now, it still won’t work. Now what??

‌

[Jonathan Doklovic]

@Shaun Martinec That is weird. After looking at the code, it seems like maybe something on your network is trying to proxy https over http. Can you please try disabling charles, and enabling the atlascode.enableHttpsTunnel flag in your settings.json?

[Shaun Martinec]

Same error: cannot get client for: Bitbucket ... Error: certificate has expired

[Jonathan Doklovic]

@Shaun Martinec would you mind running the following command and pasting the output with the actual certificates and the session stuff at the end removed?

openssl s_client -showcerts -connect bitbucket.org:443 -servername bitbucket.org

‌

[Jonathan Doklovic]

@Shaun Martinec First off, thanks for hanging in, this has been a tricky one and I appreciate your help debugging.

I’m still scratching my head as to why it works with Charles and not without it. That being said, I’ve created a new version with a hack in it that will allow you to force the axios client to use a custom cas root cert and also allow you to add a custom cert to the Bitbucket cloud calls in the same way you added one for your hosted Jira.

You can grab the new version here: https://bitbucket.org/atlassianlabs/atlascode/downloads/atlascode-2.4.11-cashack.vsix
You can install it from the extension panel in VSCode by using the “…” menu and choosing “Install from vsix…”

Once installed, restart VSCode then:

First, try just the cas root hack… open up your settings.json and add "atlascode.hackRootCas": true

If that doesn’t work by itself, then also add "atlascode.hackCertPath”:”THE SAME PATH YOU USED FOR JIRA”

Let me know if that fixes anything. Thanks!

[Shaun Martinec]

Adding "atlascode.hackRootCas": true fixed the issue. (I had to disable auto-extension updating).

[Jonathan Doklovic]

Sorry, I thought you originally said you were using a custom cert for your Jira auth. If not, then just revert your settings.json changes and I’ll do some more head scratching.

[Shaun Martinec]

I edited my initial reply and the option did work.

[Jonathan Doklovic]

@Shaun Martinec Thanks! Could I get you to try one more? I just want to make sure I know what the fix is…

here’s a new vsix https://bitbucket.org/atlassianlabs/atlascode/downloads/atlascode-2.4.11-tlshack.vsix
This does not include the hackRootCas flag anymore.

Please install this and then add "atlascode.ignoreSSLError":true to your settings.json and restart vscode.
Let me know if this works. Either way, I’ll know what the proper fix is and I can get it into the next release.

[Shaun Martinec]

That worked. I have to include "atlascode.ignoreSSLError":true in my config for it to work.

[Shaun Martinec]

Does this expose me to a man-in-the-middle attack? Any idea why the cert doesn’t work?

[Jonathan Doklovic]

@Shaun Martinec Unfortunately, this workaround is essentially setting NODE_TLS_REJECT_UNAUTHORIZED for our extension’s requests which does open up the possibility of MITM attacks.

We can’t say exactly what’s causing the error other than something in your environment seems to setting/sending the incorrect date/time when making the Bitbucket requests. I say this because we took the log you previously posted with the cert being returned by Bitbucket and dumped out the cert details. We discovered that the cert has a valid expiration but it is in a couple of weeks. So if something is setting/sending the current date and it’s off by a couple of weeks it would cause that error. We do know that currently the cert is indeed valid though.

There’s not much more we can do on our end other than the workaround above which disables the cert verification since we know a valid cert is being returned.

Hope that helps.

[Shaun Martinec]

I figured it out and the solution was easier than we thought. I had a bunch of expired certificates stored in Keychain Access. I deleted them all and now the extension (mainline) works fine without disabling SSL checking. One of them must have been an intermediate for bitbucket.org.

[Jonathan Doklovic]

@Shaun Martinec Hooray! I’m glad you figured it out! Thanks for letting us know