Push back to remote from Pipelines

Issue #13213 resolved
RichardS
created an issue

I'd like to be able to push back to the repository, maybe that's already possible but I tried some tricks and none of them worked. git push origin just gives an authentication failed error.

The use case we have is that we do deployment if a build is successful on the master branch (Tests pass, lints pass etc). Once the deployment is done we want it to tag the version with a tag to mark it as deployed, that way everyone knows exactly what commit is in what environment.

Official response

  • Graham Gatus staff

    Hi all,

    Great news - we've released the feature to allow pushing back to your repository without any additional git configuration. Please read more about the feature here.

    The git origin has been also switched from SSH to HTTP on all new accounts, and for those who haven't pushed over SSH within the last 6 weeks, allowing us to unlock new Bitbucket features in the near future. For those who are relying on the SSH origin, we will be permanently switching the origin from SSH to HTTP on February 1st 2019 - details on how to switch to HTTP or continue using SSH after this date are included in the above link.

    We hope this simplifies your CI/CD workflows with Bitbucket Pipelines.

Comments (62)

  1. Kaleb Elwert

    Any user account with write access to a repo should be able to push. The same credentials used when cloning a repo should also work for pushing.

    Currently deployment keys are read-only. If you have an integration which needs read and write access, it will need to be a full account.

    If you're having trouble with getting authenticated for git operations, your best bet would be to contact support by emailing support@bitbucket.org.

  2. Alex Soto

    I do this successfully. I create an oauth access token every build (because they expire) then set the origin:

    git remote set-url origin https://x-token-auth:${BITBUCKET_TOKEN}@bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git

    Then I git push as needed.

  3. RichardS reporter

    I guess that's a workaround but it would be neat if Pipelines could set the remote URL to some pushable URL instead (Of course only if I configure it that way, read-only as the default is of course the way to go).

  4. Ben Tatham

    We need the same thing in order to do release builds on pipelines (ie git flow release finish, specifically with mvn jgitflow:release-start jgitflow:release-finish -B). This will likely lead to extra pipeline builds on other branches, but I think once we have this feature we can work out how to suppress those.

    Right now, I have a work around using ssh access, but this has a number of issues.

    • I have to change the git origin remote as a pipeline step: - git remote set-url origin ssh://git@bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git
    • I have to use a generic, dummy user on our team to commit for everyone so I can have a single ssh key in our team's pipeline environment variables (see link above for how we do that) - I would much rather it be run as the user that runs the pipeline (either manual or commit-author based). One fix for this in pipeline might be to allow per-user environment variables that have to be set on the user account and we could put the ssh key in their, per user).

    I'm happy to discuss our work around further with someone at Atlassian, if you want to understand our use case better.

  5. Chris O'Dell

    I too would like to tag a repo after a successful build. Either through a step like this or provided as a mechanism within the Pipeline configuration. Either would work for me.

  6. Christopher Hickingbottom

    I am not able to get this to work with any of the mentioned above workarounds. I keep getting Permission denied (publickey). fatal: Could not read from remote repository.

    Ultimately, what I am trying to do is use bitbucket pipelines to run a script to build my angular app and push the contents of the dist folder to another branch. I have got it working on my local but the bitbucket pipelines doesn't.

  7. Thomas Turrell-Croft

    I don't know if this is useful to anyone but this is how you can create a tag if you are using Maven.

     custom:
        prepare-release:
          - step:
              script:
                - git config --global user.email "example@example.com"
                - git config --global user.name "Fred Bloggs"
                - mvn -batch-mode release:prepare -Dusername=$USERNAME -Dpassword=$PASSWORD
    

    The release can then be done off the resulting tag. However instead of a custom Pipeline it could be a Branch Pipeline (but then you would need to ignore the additional commit).

    The maven release plugin is doing the push over https.

  8. Sam Morrison

    I have been doing this with gitlab-ci for months. Had to switch to Bitbucket for a few reasons and am looking to do this with pipelines. This is how I managed it at gitlab (I'm not sure what changes are necessary for making work in pipelines):

      - git config user.email "example@email.com"
      - git config user.name "example"
      - git config --global url."git@gitlab.com:".insteadOf "https://gitlab.com/"
      - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
      - eval $(ssh-agent -s)
      - ssh-add <(echo "$SSH_PRIVATE_KEY")
      - mkdir -p ~/.ssh
      - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
      - [BUILD PROJECT.....]
      - git diff master ./
      - if git diff-index --quiet HEAD --; then echo "NO CHANGE"; else echo "FILE HAS CHANGED, PUSHING" && git add . && git commit -m "build-binary gitlab-CI[ci skip]" && git pull git@gitlab.com:$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME.git HEAD:master && git push git@gitlab.com:$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME.git HEAD:master; fi
    
  9. RichardS reporter

    I haven't tried it so it's just guessing on my part but the linked documentation mentions that you can use access keys on a per repo basis. The link is broken but I think it's supposed to lead here: https://confluence.atlassian.com/bitbucket/use-access-keys-294486051.html and my guess is that it works the same as access keys for users, i.e. you add the public key there and the private key to your Pipelines config and that way it gives the agent permission to write.

    You might still need to do the git config commands, to make sure that it's making commits using a predictable name and email which might be nice for history.

  10. RichardS reporter

    Haha, well that's interesting. Then I guess you would have to add the key to a bot user which you would then give access to your repository.

  11. Jeffrey Labonski

    I can confirm that Alex Soto's solution works. Expose an OAuth2 consumer with write access. Here's a quick bash script I whipped up that I call from pipelines with the OAuth key and secret passed in via environment variables.

    It would be much, much better to have the option to elevate to r/w access explicitly in pipelines rather than having to do this rigamarole.

    if [ -z $1 ]; then
        echo "ERROR: No key"
        exit 1
    fi
    
    if [ -z $2 ]; then
        echo "ERROR: No secret"
        exit 1
    fi
    
    KEY=$1
    SECRET=$2
    
    set -e
    
    wget -q -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
    chmod 755 jq
    TOKEN=`curl -s -S -f -X POST -u "${KEY}:${SECRET}" https://bitbucket.org/site/oauth2/access_token -d grant_type=client_credentials | ./jq .access_token -r`
    
    git remote set-url origin https://x-token-auth:${TOKEN}@bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git
    # git is now pushable
    
  12. John Persson

    Anyone got any updates on this one? If not, what would be the best current workaround in your opinion?

    Been stuck trying to solve this very same problem when working on incorporate semantic-release to automate the releases and version tagging of our NPM packages.

  13. Davy De Waele

    I tried the username / password approach by Thomas Turrell-Croft but couldn't get it to work.

    [ERROR] The git-push command failed. [ERROR] Command output: [ERROR] fatal: repository 'https://$USERNAME:$PASSWORD@bitbucket.org/davydewaele/springboot.crud.sample/' not found [ERROR] -> [Help 1] [ERROR]

    the only way I'm able to tag / push to the remote from a build pipeline is to

    • use ssh instead of https
    • create a key-pair in the bitbucket pipeline settings
    • configure the public key from the bitbucket pipeline SSH key (from the step above) on the bitbucket account hosting the repo.
  14. Davy De Waele

    Hi Thomas, I am using your sample project (thanks by the way !) and building on top of it for a more elaborate git flow pipeline.

    I simply noticed I couldn't get it to work with username / password authentication. Perhaps related to the recent changes Altassian did with their login flow. (We're now authenticating using Google. Perhaps related to that). But replacing HTTPS with SSH based authntication works fine on your project. You just need to setup some keys in Bitbucket.

    We still have a number of maven projects using the dinosaur maven release plugin that doesn't get a lot of love anymore but still "kinda" does the job. So your project proved to be a nice starting point.

  15. John persson

    @Dieter De Waele I successfuly used the user/password approach from Jeffreys example.

    However if the authentication process has changed recently I can't tell if we still have a working solution.

    We need authentication as a step in automating releases to NPM. We're planing a release next week, I will keep you updated.

    I use Travis CI with github for my personal open source projects, which works great without extra steps for authentication. I was surprised Pipelines hasn't adopted the same behavior by default.

  16. John Persson

    I can confirm the user/password approash is still working.

    This is my script for auth, as I mentioned previously it's taken from @Jeffrey Labonski

    # Script for granting git access to the CI of the current repository on Bitbucket
    # This script expect the enviroment variables $OAUTH_KEY and $OAUTH_SECRET, to be pressent in the bitbucket
    
    if [ -z $OAUTH_KEY ]; then
        echo "ERROR: No key"
        exit 1
    fi
    
    if [ -z $OAUTH_SECRET ]; then
        echo "ERROR: No secret"
        exit 1
    fi
    
    KEY=$OAUTH_KEY
    SECRET=$OAUTH_SECRET
    
    set -e
    
    wget -q -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
    chmod 755 jq
    TOKEN=`curl -s -S -f -X POST -u "${KEY}:${SECRET}" https://bitbucket.org/site/oauth2/access_token -d grant_type=client_credentials | ./jq .access_token -r`
    
    git remote set-url origin https://x-token-auth:${TOKEN}@bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git
    # git is now pushable
    

    Then inside my pipelines script I create a git user for the CI and run the above script like so:

        master:
          - step:
              caches:
                - node
              script:
                - chmod +x ./pipelines/oAuthGit.sh
                - ./pipelines/oAuthGit.sh
                - git init
                - git config user.name "CI User"
                - git config user.email "main@exampel.com"
    

    @Dieter De Waele Make sure you set the required environment variables of the repo.

  17. John Persson

    You define these manually in the Environment Variables of your repo.

    Under: settings / Environment Variables

    The variables defined here are made available inside of pipelines.

  18. Andrew Katsivas

    Not sure if this will solve anyone else's problem, but for me I did the following (based on a ticket on the Atlassian support):

    1. Generate the SSH Key under Pipelines
    2. Copy the public key
    3. Open up your User Settings
    4. Under Security > SSH, add the public key to your account

    Pipelines script can now automagically push back to the repo without any special scripts or other config.

    Hope this helps someone!

  19. Alvaro Del Valle

    ⏫ This worked for me. I added the pub key to my team settings' SSH keys. Then in my bitbucket-pipelines.yml file I added a config to push a tag just how I intended to do. - git config --global push.default simple && git config --global user.email "USER@COMPANY.com" - git config --global user.name "CI Built" && git add . && git commit -m "testing [skip ci]" - git push --follow-tags origin master

  20. Rohit Vishnu

    This worked, put i do not see the new commit in my repo.

    i did the below.

    git add test.iar git config user.email xxx@xxte.com git config user.name rohitvishnu git commit -m "This is a test" git push origin master

  21. Mihai Varga

    Hello,

    tried above with flow recommended by Atlassian support. However I get the following:

    Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

    Can you advise please?

  22. Oleg Sigida

    @Mihai Varga To push from Bitbucket Pipeline is the same how to push from any other machine or laptop. So you need to full fill 4 thing:

    1. You need to have a user who has access to the repository. See: https://confluence.atlassian.com/bitbucket/create-and-grant-access-to-team-repositories-665225545.html

    2. The user should have ssh key defined. See: https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html

    3. This ssh key should be set to BitBucket Pipeline. See: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

    4. Set the user name and mail when you commit something to the repo. You need to add follow steps to the pipeline, before you make a commit

      - git config user.email "user@email.com"
      - git config user.name "user"
    

    It is difficult to say where you have an issue as the error just state that the user who tries to push does not have access to the repo.

    Hope this will help.

    PS: I don't see this as an error, and believe the access to the repo from pipeline should be granted explicitly.

  23. Mihai Varga

    So what I did was this:

    1. Generate the SSH Key under Pipelines
    2. Copy the public key
    3. Open up your User Settings
    4. Under Security > SSH, add the public key to your account

    I am an admin on the repository I am talking about. It there something I should be doing differently?

    The weird part all worked fine like above last week until Friday when it stopped working and throwing that error.

  24. Keynan Pratt

    @Mihai Dan Nadăș iirc that key is only granted read access, not write access so you should be able to read, but not write / push

    @Oleg While that might technically work it is a hack, not a reasonable solution. It either requires A) 1 bot per repo, or B) a bot with excessive access. The former being a hack, and the latter being a security no-no

  25. Mihai Varga

    @Keynan Pratt so you are basically saying this cannot be fixed for now :) ?

    The weird part is all worked fine until last week. I generated a new SSH key on the pipeline because I tied it to a new account and all stopped working. Like the key in the pipeline does not match the one on the settings. I can confirm it does not after looking into the ~/.ssh folder. I cannot set the old public key back since Bitbucket does not provide access to the private key. Sounds like a weird bug on the pipelines.

  26. Oleg Sigida

    @Mihai Varga I suggest you to start over.

    Generate new pair of private and public keys, and keep them locally.

    Update your user with the public key (who has access to the repo), then update SSH setting for the pipeline with keys (you would need both public and private keys there).

    It won't work until all they keys match.

    What @Keynan Pratt is saying that this flow is either to difficult too maintain or leads to potential security issues.

  27. Pablo Compagni

    I've just set this up and it's not as hard as it seems. On the repo settings, under pipeline, you can set the ssh key you want the pipeline to run with. Then it's a matter of that key having write access to the repo. Either adding it to a user, or to the team itself.

  28. Mike Chaberski

    The issue with adding the new key to your user SSH keys is that anyone with write access to the repo can gain access to the private key, and therefore can gain access to any repository the user has access to. (They could do this by changing the pipeline script to print the private key.)

    Creating a separate user is a workaround, with the caveat that the exposure applies to the separate user's repositories. But this is all immaterial if you can't create a separate user, as is the case if the user needs to be a member of a corporate team. It would be more helpful if the pipeline SSH key had limited write access to the same repository.

    The "Build Setup" stage has an oauth token that it uses to clone the repo, defined as the value of the REPOSITORY_OAUTH_ACCESS_TOKEN environment variable. It would be nice if that token provided write permission limited to only the current repository and the environment variable remained defined in the pipeline step.

  29. Matus_bitaccess

    This is a bummer, I finally found it after 1 & 1/2 day of searching to make this work, after the tests pass on develop push to master would then trigger next build and deployment on staging. I was using this on shippable, it would be really helpful to have git push write access.

  30. Raul Gomis staff

    Hi everyone,

    Thanks for all of the feedback on this issue. Some good news: we've started speccing out a feature that will let you push back to repository without having to set up SSH keys or any other workarounds.

    I'll keep you posted with any updates.

    Thanks for helping us improve Pipelines!

    Regards,
    Raul

  31. Ben Tatham

    Glad to hear! It would of course be great if the "push" to git were done as the user that started the pipeline...ie the user who made the commit that triggered it, the user that pushed the manual button, etc. That is what is missing right not...we have done all the workarounds, but that means all pushes from pipelines are done by the same user (at least the way we have it setup).

  32. Raul Gomis staff

    Hi everyone,

    A quick update on this ticket: we are currently working on it and we expect to release it before the end of the year if we don't find major issues.

    I'll keep you posted with any more updates in the next week(s).

    Regards,
    Raul

  33. Graham Gatus staff

    Hi all,

    Great news - we've released the feature to allow pushing back to your repository without any additional git configuration. Please read more about the feature here.

    The git origin has been also switched from SSH to HTTP on all new accounts, and for those who haven't pushed over SSH within the last 6 weeks, allowing us to unlock new Bitbucket features in the near future. For those who are relying on the SSH origin, we will be permanently switching the origin from SSH to HTTP on February 1st 2019 - details on how to switch to HTTP or continue using SSH after this date are included in the above link.

    We hope this simplifies your CI/CD workflows with Bitbucket Pipelines.

  34. Log in to comment