curl with --digest not downloading from repos today. (BB-12465)

Issue #11160 resolved
izzy m created an issue

This is a new problem. It was working fine before today. Using curl to fetch from repos is not downloading anything. curl --digest -u "user" -o master.tar.gz

Comments (21)

  1. izzy m reporter

    It looks as though the download url must use the commit number to succeed. Has bitbucket changed the handling of download url for branch head? This is a major problem for useability.

  2. izzy m reporter

    Leaving out the --digest option allows it to work. Since the url is an https url and ssl/tls is being used this is probably secure. Any thoughts?

  3. Dominick Staniforth

    I found the same, but I couldn't find anything definitive about the security being acceptable and I'm not sure it's safe, so I'm not using this for now. Most searches on the subject were related to the drawbacks of digest being mostly negated by SSL. I've raised a support ticked about this. I'll put anything useful/relevant they respond, here.

  4. izzy m reporter

    Thanks for the response.

    I only happened to discover the solution to the problem when i explored curl --trace and looking at the output file it appears as if all the handshaking is taking place.

    Reading the curl manual it may be that --digest was meant for an http connection not an https connection. Maybe until now --digest worked when used with https but isn't really needed because the url is declaring secure protocol.

    I haven't explored the curl issues. Really had other tasks at hand when this happened.

  5. Jesse Yowell Account Deactivated

    Thanks guys -- I'm raising this as an internal ticket to see why we've removed support for the --digest flag using curl

  6. Jesse Yowell Account Deactivated

    Looks like we nuked support for --digest recently, as many of you have confirmed: it's not really necessary since we already use HTTPS.

  7. Jesse Yowell Account Deactivated

    I'm resolving this since the workaround is just removing --digest from the curl command. Unless there is a really good reason for adding back support for it, I think we can close this.

  8. izzy m reporter

    Jesse, thanks for your involvement.

    If you mean that since there is a work around you have simply changed the status to resolved, please allow me to differ.

    First, if this has resulted from some change on bitbucket it may be useful to examine what that is and to highlight for others what has given rise to this now failing.

    1. The change is not backwards compatible and will cause existing scripts and work flows to fail. Had we not been actively working on this we may have missed isolating the cause and been caught by surprised with failures extending from this.

    2. The instructions most likely found by users doing internet searches includes the --digest option. To have it fail will not be helpful or kind to new users.

    3. Unless there are compelling reasons this should be changed back. If curl can ignore the --digest option when it's not truly needed there is no real harm. People can learn more gradually that it is not required when other specifications are met. Using it may be seen as a fail safe.

    Please confirm this resulted from configuration changes at bitbucket and share with others how to avoid the same issue. Also please consider reverting the change or explaining why that is not a good idea.

    Thank you.

  9. izzy m reporter

    I have changed the status of this issue to open.

    Questions were raised by Dominick in the third comment as to whether this change is secure, acceptable and safe.

    And there are several points I raised in the last comment.

    These should be addressed before this issue can be considered resolved.


  10. Nate Dudenhoeffer

    I would like second what izzy said about reverting this change. I can confirm that removing the digest option works.

    Why the backwards incompatible change? A quick google search will show that many people use this method for pulling code to their servers.

  11. matthiasDrummer

    I agree with izzy. I ran yesterday into this issue by accident. Luckily only our staging environment was affected and I was able to the change the deployment scripts (removed the --digest option) and EC2 images today on the production environment before any downtime occured.

  12. Erik van Zijst

    Yes, this was a backwards incompatible change that we shouldn't have made. We've reverted it for now and I'm sorry for the inconvenience.

    We messed up in not providing upfront notice, but support for Digest is something we are actively looking to drop. As pointed out by several people on this issue, Digest offers advantages over Basic HTTP Auth when using insecure channels. This is also where our Digest support finds its roots. It was a useful alternative auth method back when we still offered non-SSL access.

    Today however, Digest offers nothing extra except added complexity and, in the case of curl, an additional round trip to request a challenge.

    Note that for years our remaining Digest support has been limited to a handful of URLs and we're planning on permanently dropping it in the future. However, we will issue an announcement.

  13. izzy m reporter

    Erik, Thank you for the update and further information on digest. It certainly helps to clarify the issue. If this information can be placed in the Bitbucket 101 documentation it will help increase awareness that digest is not needed and allow for an easier transition in the future when it is dropped. Knowing that use of digest is not needed with SSL and adds a performance hit is helpful. Thank you.

  14. Log in to comment