Generated URLs do not work with our server

[Jens Hatlak]

Bitbucket Linky 3.4 PHPStorm 2017.1.3 Bitbucket v4.14.4

The URLs Bitbucket Linky generates do not work with our server.

Pull Request

Linky-generated: https://<server>/<project>/<repo>/pull-requests/new?source=<sourceBranch>&dest=<targetBranch>

Correct: https://<server>/projects/<project>/repos/<repo>/pull-requests?create&sourceBranch=<sourceBranch>&targetBranch=<targetBranch>

Source

Linky-generated: https://<server>/<project>/<repo>/src/<commit><fullFilePath>

Correct: ?

[Daniil Penkin]

Hello @jhatlak,

Thanks for using Linky and reaching out!

I believe that Linky generates URL for Bitbucket Cloud instead of Server. There're few possible reasons for that:

1. Linky has a bug and discovered Bitbucket Server as Cloud, therefore the links are wrong.
2. Linky checks remotes of your repository (btw, is it Git or Hg?), so it might be also that you have more than one remote, where one is pointing to Server, and another one to Cloud.
3. You've linked the repository manually and misconfigured the link. Is that the case?

So I'd start with the following question: did you configure the link or Linky did everything automatically?

Looking forward to hear from you and to fix your issue.

Cheers, Daniil

[Jens Hatlak]

Yeah I guess the detection is wrong; I only have option "Cloud" in the "Fixed" drop-down. I use option "Automatic", though. The plugin was previously working for me but I cannot tell when it broke. Possibly through one of the version updates. We're using Git.

I don't think I misconfigured the link but I cannot really tell. I cannot change the settings anymore, can I? How can I reset the settings and start from scratch (only concerning Bitbucket Linky of course)? Would uninstalling/reinstalling the plugin work?

[Daniil Penkin]

I meant another settings screen which you don't see now (since link has been created already) – for initial link creation, – but if you say that Linky worked before, then it's unlikely the case.

I cannot change the settings anymore, can I?

No, once you configured a link, there's no way in the Linky's UI to remove or change it. I have issue #23 which is exactly about that, and I'm still thinking whether this should be present in the UI.

How can I reset the settings and start from scratch (only concerning Bitbucket Linky of course)? Would uninstalling/reinstalling the plugin work?

Unfortunately, reinstalling won't work: configuration is not removed, and I'm not sure there's a way to do that. I have a task in the backlog to check if there's a way in IntelliJ API to cleanup the configuration when the plugin is removed.

So what you can do now is modify or delete Linky configuration files. Let's reset Linky and see if that helps.

You didn't mention which OS you are using, so I'll use paths for macOS; for Linux and Windows they are same inside IntelliJ's config directory, but its location is different.

So, here are the steps you should take:

• Close PHPStorm. Otherwise the settings you change might be overridden to the old values when the IDE gets focus back or when it quits.
• Navigate to /Users/<your_user>/Library/Preferences/<ide_with_version>/options/, e.g. in my case it's /Users/dpenkin/Library/Preferences/IntelliJIdea2017.1/options/.
• There will be two files that Linky uses: bitbucket-cloud-registry.xml and bitbucket-server-registry.xml which keep the registry of discovered Bitbucket Cloud and Bitbucket Server hostings, respectively.
• Delete these two files — this will effectively reset Linky.
• Now you can reopen Bitbucket-hosted repository in PHPStorm. Linky will try to discover Bitbucket, and if it fails to do so (for instance, Bitbucket is served on non-standard HTTP port or you use SSH aliases etc.), you'll be able to configure the link manually in Settings:

If you end up configuring the link manually, be careful when selecting the value in that Server/Cloud dropdown: this is what I meant in my previous comment, and what tells Linky the type of links it should generate.

Let me know if that worked.

Cheers, Daniil

[Jens Hatlak]

Removed the files. They were recreated automatically as before. So I guess the automatic determination is buggy. If it helps, we use scheme ssh and a (custom?) port for Git. I'm on Windows.

[Daniil Penkin]

Ok, this sheds some light on what's happening.

Could you please confirm two things:

• wrong record has been created in the re-appeared bitbucket-cloud-registry.xml (that said, you can find the base URL of your Bitbucket Server in that file)
• bitbucket-server-registry.xml file doesn't contain any actual records

Thanks for your help with the investigation!

Hopefully tomorrow I'll have some time to take a look at the code.

Cheers, Daniil

[Jens Hatlak]

Both confirmed. Hope you find the issue - since there is currently no way to do a manual setup, I'm stuck.

[Daniil Penkin]

Thank you @jhatlak.

Will see if I'm able to reproduce this.

[Daniil Penkin]

Hello @jhatlak,

Sorry for such a long delay.

Unfortunately I can't reproduce the issue you faced. I suppose there's some name collision but I can't explain how REST check, which Linky performs as part of the discovery procedure, confuses Server and Cloud: test endpoints and responses are very different. Could you please help me further with the investigation?

I would like to ask you to collect a debug log that will show how Linky performs Bitbucket discovery. Here're the steps to do that:

1. In PHPStorm navigate to Help → Debug Log Settings... and add BitbucketReferences category — this will make Linky log more stuff about Bitbucket discovery.

   

2. Save the form and close PHPStorm.

3. Remove same bitbucket-cloud-registry.xml and bitbucket-server-registry.xml files, so that Linky will make an attempt to re-discover Bitbucket next time you open the project in IDE.
4. Reopen PHPStorm with the affected project, wait a minute or so until it resolves Git status etc.

5. Navigate to Help → Show Log (in Finder in my case) to locate the log file.

   

6. Attach that file to this issue or, in case you have security concerns, you can encrypt it with my key and email to me to dpenkin [at] atlassian.com, or just post it here encrypted, or let me know which other communication channel is the most convenient for you.

Thank you in advance!

Cheers, Daniil

[Jens Hatlak]

Set up debugging as requested and checked the logs. It seems Bitbucket Linky detects our address as matching both Cloud and Server but then goes on to just check the Cloud part - first with https and standard port (result: 404), then with https and port from the Git SSH remote address (result: Unrecognized SSL message), then with http.

#!

2017-05-22 16:02:59,117 [   8850]  DEBUG -            BitbucketReferences - Remote URL path '/<our path>' matches Bitbucket Cloud repository path pattern 
2017-05-22 16:02:59,144 [   8877]  DEBUG -            BitbucketReferences - Remote URL path '/<our path>' matches Bitbucket Server repository path pattern 
2017-05-22 16:02:59,161 [   8894]  DEBUG -            BitbucketReferences - Remote URL path '/<our path>' matches Bitbucket Cloud repository path pattern 
2017-05-22 16:02:59,169 [   8902]  DEBUG -            BitbucketReferences - Probing for Bitbucket Cloud at 'https://<our host>/ 
2017-05-22 16:02:59,493 [   9226]  DEBUG -            BitbucketReferences - Error while checking Bitbucket Cloud candidate url 'https://<our host>/ 
java.lang.RuntimeException: retrofit2.adapter.rxjava.HttpException: HTTP 404 Not Found
(...)
2017-05-22 16:02:59,495 [   9228]  DEBUG -            BitbucketReferences - Probing for Bitbucket Cloud at 'https://<our host>:<our port>/ 
2017-05-22 16:02:59,534 [   9267]  DEBUG -            BitbucketReferences - Error while checking Bitbucket Cloud candidate url 'https://<our host>:<our port>/ 
java.lang.RuntimeException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
(...)
2017-05-22 16:02:59,535 [   9268]  DEBUG -            BitbucketReferences - Probing for Bitbucket Cloud at 'http://<our host>/ 

I guess the problem is that it tries Cloud first and thinks http is usable, and does not try Server after that.

Where is the code determining which paths match Cloud and/or Server?

[Daniil Penkin]

Path matching is only the first filter which helps to avoid useless network requests (if the path doesn't match, it's 100% not Bitbucket host). Patterns are simple: here's one for Cloud, and here're two for Server (http and ssh URLs have different patterns for Server). So as you can see, URL patterns for ssh are actually same, and it's totally ok that both Cloud and Server matched.

The next step Linky makes – which you can see in the log – is it goes over possible combinations of scheme (http and https) and port (default and discovered one) and tries to request known REST endpoints for each of that combination.

Firstly, it constructs possible base URLs. For instance, if git remote URL of the repository is ssh://[e-mail]:7999/some/path/project/repo.git, Linky will try following base URLs (as it doesn't know where exactly Bitbucket runs):

• http://some-bitbucket-host.com/some/path (i.e. port 80)
• http://some-bitbucket-host.com:7999/some/path
• https://some-bitbucket-host.com/some/path (i.e. port 443)
• https://some-bitbucket-host.com:7999/some/path

Similarly, for http/https git remotes Linky will try only some of these base URLs, assuming that the interface runs on with the same scheme as git server.

Then it tries to request known public REST endpoints to verify that it found Bitbucket.

• For Cloud, it checks http(s)://api.bitbucket.org/2.0/hook_events or http(s)://some-other-bitbucket-cloud-host/!api/2.0/hook_events, depending on exact host name.
• For Server, it tries http(s)://host:port/rest/api/1.0/application-properties.

The thing is that although response is not used, it is parsed, so if the hosting doesn't reply with the expected JSON format, it will be considered as not-a-Bitbucket hosting, and Linky won't use it to construct links.

Linky scans all remotes and all possible URLs constructed as I described until it gets a correct response from the REST endpoint (it's even a bit more complicated, but in your case it's just one discovered hosting, so let's not go deeper). Once it got a response, it assumes that Bitbucket instance was found, and uses it to construct links for you.

Now, the question is why does Linky decide that it found Bitbucket Cloud in your case. The log excerpt you posted doesn't show all probes. Are there any messages "Probing for Bitbucket Cloud at ..." that don't have matching "Error while checking Bitbucket Cloud candidate url '...'"? I assume there's at least one.

[Daniil Penkin]

I re-read your last message and realized I probably misunderstood you.

So if that's the whole log, not an excerpt, Linky has decided that 'http://<our host>/' is Bitbucket Cloud. That would mean that the following REST endpoint returned something: http://<our host>/!api/2.0/hook_events.

Can you confirm or deny this assumption e.g. by checking with curl? AFAIK, BBS shouldn't have anything on that endpoint.

[Jens Hatlak]

For that URL I get 301 Moved Permanently which is a redirect to https://<our host>.

[Jens Hatlak]

Any news, or further information you need from me? I could imagine the code is confused by the 301 return code.

[Daniil Penkin]

Hi @jhatlak,

I'm very sorry to drag this out, I didn't have much time during last couple of weeks :(

Yes, I agree that redirect is somehow involved but I wasn't able to reproduce this behaviour on my local setup and make a fix before I had to park my work on Linky for a while. I was going to prepare a build with verbose debug logging for you to try, if you don't mind.

[Daniil Penkin]

Hello @jhatlak,

I'm very sorry for three months of silence, parenthood is quite time consuming :) Hopefully I'm back to business now.

Unfortunately, I wasn't able to reproduce the issue with different kinds of configurations I tried, including redirects. However, version 3.5 that I've just released has somewhat improved low level request logging which might help with the investigation. Could you please try the following:

• enable BitbucketReferences DEBUG log category, as described above
• locate idea.log file
• close the IDE
• remove or clear idea.log file that you located earlier to make it easier to inspect results of this experiment later
• remove following files (or move them out temporarily if you want to preserve all Bitbucket hosts that Linky discovered since installed) — this will cause Linky to re-discover remote repositories for each Bitbucket-like local repos; in my case IDE_PREFERENCES_DIR is /Users/dpenkin/Library/Preferences/IntelliJIdea2017.2:
  • <IDE_PREFERENCES_DIR>/options/bitbucket-cloud-registry.xml
  • <IDE_PREFERENCES_DIR>/options/bitbucket-server-registry.xml
• reopen the affected project and give Linky some time to run the discovery process — a couple of minutes should be enough
• close the IDE
• idea.log should now contain low level logs of HTTP requests that Linky issues during discovery; they're logged under different categories, containing BitbucketCloudApiImpl and BitbucketServerApiImpl, however their log level depends on BitbucketReferences
• attach the idea.log file to this ticket or, if you have security concerns, GPG-encrypt it using my public key and attach the encrypted version, or send it to my email dpenkin (at) atlassian.com or simply paste the encrypted text in a comment here — no matter how you do that, only I will be able to decrypt and read that text.

This would be very helpful. Let me know if you have any questions.

Thank you very much for your patience and understanding.

Cheers, Daniil

[Daniil Penkin]

Hi @jhatlak,

I now clearly see where the problem is and will work out a fix soon. Thank you for your help with the investigation!

Cheers, Daniil

[Daniil Penkin]

Hello @jhatlak,

I've just released Bitbucket Linky 3.6 which should handle your environment config properly. Could you please try it out and let me know if the fix works?

You'll need to tweak or remove *-registry.xml files again, otherwise Linky will rely on the incorrect discovery results from the past.

Thanks a lot for your patience.

Cheers, Daniil

[Jens Hatlak]

After the reset (removing the two files and restarting) I was able to complete the setup by manually entering the Bitbucket Server URL via the Settings UI now. If that is the expected behavior (as opposed to automatic detection) then all is fine now.

[Daniil Penkin]

Hi @jhatlak,

I'm glad to hear that Linky now works for you. Thanks a lot for your patience and willingness to help with the investigation!

If that is the expected behavior

Well, generally Linky knows how to detect Bitbucket Server instances, but there're cases when it can't do that. Linky uses the /rest/api/1.0/application-properties endpoint to check whether the host is Bitbucket Server or not. So if that endpoint doesn't return what Linky expects it to return, automatic discovery ignores it.

Can you check this endpoint in your browser, e.g. try opening https://<bitbucket_host_and_port>/rest/api/1.0/application-properties (important update: please open it in the incognito mode to ensure you're not logged in to better simulate what Linky requests) – if it returns something like this:

{
    version: "5.5.0-rc2",
    buildNumber: "5005000",
    buildDate: "1508117586992",
    displayName: "Bitbucket"
}

then Linky likely has a bug. But if this endpoint is not available or it redirects you to somewhere else, then it is an expected Linky behaviour.

Please let me know if it works, I'm keen to fix another bug if it's there.

Cheers, Daniil

[Daniil Penkin]

Removing version: 3.4 (automated comment)