fetch command breaks because of reliance on user-agent (BB-11638)

Issue #10660 resolved
Anonymous created an issue

BitBucket doesn't seem to like the FreeBSD fetch command authenticating and downloading files from a private repo. This makes it difficult to download a post-install script on a fresh installation (where wget is not included by default).

However, changing the fetch user-agent string from "fetch libfetch/2.0" to "Wget/1.16 (freebsd10.0)" does allow authentication. So perhaps relying on user-agent strings is not the way to go.

The following does not work:

~# fetch --no-verify-peer "https://myname:mypass@bitbucket.org/me/server/raw/abcde...xyz/post-install.sh"

However, the following does work:

~# fetch --no-verify-peer --user-agent "Wget/1.16 (freebsd10.0)" "https://myname:mypass@bitbucket.org/me/server/raw/abcde...xyz/post-install.sh"

Comments (3)

  1. Erik van Zijst staff

    What happens here is Bitbucket trying to decide whether the client is a browser or not.

    It does this because when hitting a private resource while not being logged in, Bitbucket will redirect a browser to the login page. However, when the client is not a browser, then that is not terribly useful. Instead, we should send a 401 response so the client can ask the user for credentials and retry.

    This is where curl and fetch start to deviate.

    When providing credentials as part of the command (as you are doing), curl will eagerly send them along with the request. As a result, the request is never anonymous and accessing the private resource succeeds.

    On the other hand, BSD's fetch does not send credentials preemptively, even if provided as part of the command. Instead, it waits for the server to send a 401 response with a challenge. Only then will it retry. However, as Bitbucket doesn't know that fetch is not a browser, it instead sends a 301, breaking the client.

    I have put in place a hardwired rule for libfetch so that it won't get redirected to the login page. That change should get deployed later this week.

  2. Log in to comment