Pipelines - id_rsa not found since this morning (Deployment process broken)

Issue #14729 resolved
Robin Rijkeboer created an issue

Since today, whenever I run a pipelines build this happens:

scp build.tar.gz user@$ACC_SERVER_IP:~/
no such identity: /opt/atlassian/pipelines/agent/data/id_rsa: No such file or directory
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).

Pipelines are an essential part in our deployment proccess and we seem to hit this error on all our pipelines deploys.

Here is a community thread where people also encountered this issue since this morning: https://community.atlassian.com/t5/Bitbucket-questions/Could-opt-atlassian-pipelines-agent-data-id-rsa-become-invalid/qaq-p/631206#U631240

Official response

  • Samuel Tannous staff

    Hi everyone,

    We've spent the past couple of days investigating how the original change could have caused this issue. Thank you to the customers who assisted in this investigation. Unfortunately we were unable to reproduce the problem including in any of the customer repositories who experienced the issue and granted us permission to debug directly.

    The original code change was intended to enable SSH key support for non-root users. We have rearchitected the solution to reduce the risk to existing configurations but as we were unable to identify root cause there remains a small chance that this change may affect some users. We are prepared to work with those users and revert the change if necessary.

    Regards Sam

Comments (50)

  1. James Quinton

    We're having the same issue.

    No change to our keys, however we also get the error:

    no such identity: /opt/atlassian/pipelines/agent/data/id_rsa: No such file or directory

  2. Ronald Chia staff

    As a workaround while our team is investigating this issue, you can try adding the following steps in your script, which load the SSH-key to ssh-agent, and you should then be able to run your builds

              - eval `ssh-agent`
              - ssh-add /opt/atlassian/pipelines/agent/data/id_rsa
    
  3. Peresson Maxime

    Doesnt work for me either

    + ssh-add /opt/atlassian/pipelines/agent/data/id_rsa
    /opt/atlassian/pipelines/agent/data/id_rsa: No such file or director
    
  4. Robin Rijkeboer reporter

    @ronaldchia The workaround seems to work for us. Thank you. Any estimate to when this issue would be fixed?

  5. Matthew Drouin

    Doesn't work for us.

    + ssh-add /opt/atlassian/pipelines/agent/data/id_rsa
    /opt/atlassian/pipelines/agent/data/id_rsa: No such file or directory
    

    Is there error we get.

  6. Billel Abidi

    It does not work for me:

    + ssh-add /opt/atlassian/pipelines/agent/data/id_rsa
    /opt/atlassian/pipelines/agent/data/id_rsa: No such file or directory
    
  7. Matthew Drouin

    I created a release script that did nothing but ls -alh /opt/atlassian/pipelines/agent/data and I noticed the file was there @ronaldchia from support emailed me the suggestion. I moved the recommended solution to the top of my scripts, first things that gets in the script tag, and everything seems to work now.

    I was running it after all my tests and builds previously and that seems to have been causing issues.

  8. Gary Sackett staff

    Hi Everyone, Our team is actively working towards a permanent fix for the issue. We will update this issue with a more firm eta soon.

    Cheers, Gary

  9. Philip Hodder staff

    Hi everyone,

    We've deployed a fix for this issue and are investigating if this is fully resolved.

    Can you please let us know if the issue is resolved or still occurring for you.

    Thanks, Phil

  10. Abdel Atencio

    still having issues. no such identity: /opt/atlassian/pipelines/agent/data/id_rsa: No such file or directory

  11. Philip Hodder staff

    If you're still experiencing this issue, can you please contact support at https://support.atlassian.com/ so we can continue to investigate this. Please provide an example repository/pipeline which is encountering this problem.

    Thanks,

    Phil

  12. Tim Brandin

    We're still having issues, but it might be a new one, this time it has to do with Host key verification.

    The key exists in the container, I can verify that and I believe it is the right one, and that key is associated with a user with access to the repo we are trying to merge into, but still I get: "Host key verification failed.". And I've tested to add bitbucket in known hosts (even though it says one should need to ad bitbucket in the docs) without any luck.

  13. Tim Brandin

    I've gotten help to resolve our issue with Host key verification, somehow the change yesterday made it so that known_hosts was not available anymore to the user we use in our docker image, which means we have to build up the known_hosts ourself and cannot rely on the interface in bitbucket pipelines ssh key settings.

  14. Philip Hodder staff

    Hi Tim,

    We found the cause of your issue with known_hosts. We'll update this ticket again when we've released the fix to production.

    Thanks,

    Phil

  15. Tero Pesonen

    Our pipeline build started working again yesterday afternoon and has worked since that, something has been fixed I suppose. Thank you for support!

  16. Philip Hodder staff

    We've deployed the known_hosts fix to production. Let us know if it's working as intended now. :)

    Thanks,

    Phil

  17. Samuel Tannous staff

    Hi everyone,

    We've spent the past couple of days investigating how the original change could have caused this issue. Thank you to the customers who assisted in this investigation. Unfortunately we were unable to reproduce the problem including in any of the customer repositories who experienced the issue and granted us permission to debug directly.

    The original code change was intended to enable SSH key support for non-root users. We have rearchitected the solution to reduce the risk to existing configurations but as we were unable to identify root cause there remains a small chance that this change may affect some users. We are prepared to work with those users and revert the change if necessary.

    Regards Sam

  18. Log in to comment