Allow more capable steps that can be reused

Issue #12751 closed
Arjen Schwarz
created an issue

At the moment all the steps are limited to a list of commands that can be run. While useful, a more powerful solution would be to allow other types of scripts. An example of this would be an AWS Lambda step where you only have to provide your settings and everything else is already done for you. Currently you need to go through several steps, which includes adding a script to your repository, and that's not an optimal solution.

An example of how this works in a different CI tool is this step that I created for Lambda deployments. Especially the ability to share steps like this across your projects (or with other people), would make Pipelines even more powerful.

Obviously, this would depend on #12750

Official response

  • Aneita Yang staff

    Hi everyone,

    I've created two new tickets to track the main requests that we've identified on this issue. With this current issue, it has become difficult to understand who is interested in what, and which request is more important to our users - the two main asks that we've identified are slightly different projects for us.

    • Issue #17182 now tracks the request to have better pre-defined templates to help configure Pipelines to work with other tools (e.g. deployments to AWS).
    • Issue #17181 tracks the ability for users to define their own configuration which can then be reused across multiple repositories.
    • If you are interested in being able to reuse configuration within the same YML file, you can check out YML references and anchors.

    Please vote for and watch the new tickets linked above based on what you're interested in seeing. This will help us gauge the importance of each request. If you have a suggestion that's not covered by the two tickets above, please raise a new ticket so that we can assess and track it separately.

    Thanks for the help!

    Aneita

Comments (46)

  1. Julio César Canares García

    I got the issue were I need to run the test step in all branches and also having specific steps in some branches, something like in gitlab-ci

    stages:
      - build
      - test
      - release
    
    build:
      stage: build
      script:
        - echo foo
    
    unit-test:
      stage: test
      script:
        - echo foo
    
    code-quality-test:
      stage: test
      script:
        - echo foo
    
    release:
      stage: release
      script:
        - echo foo
      only:
        - master
    
  2. Jason Collinge

    Being able to share scripts between steps/branches is pretty important for making efficient and maintainable pipelines. For instance, the build steps for a staging and production branch need to be the same, but right now you have to duplicate code between the steps on both branches. This quickly gets long and ugly and increases the likelihood of having differences in the common steps creep in. Being able to share steps between different pipelines would also be useful as the number of pipelines grows (similar to templates in gocd) but I don't think it's as critical.

  3. Matt Ryall staff

    Thanks for the feedback on this issue. We're starting early design work on this feature, with development due to start around March-April 2018, after a few higher priority items are completed.

    If you have some time to discuss your needs for reusable steps in Pipelines, please shoot me an email at mryall@atlassian.com, and we can line up a time to chat.

    Please note that there are several existing workarounds to reusing step logic in Pipelines that I'll note on this ticket in case people aren't aware of them:

    • write a Python/Ruby/Node/shell script in your repository, and call it from different places in your YAML configuration
    • create a custom Docker image, which can set up any necessary build prerequisites and bundle additional scripts
    • use YAML references to replicate some of the step logic in the same config file (painful but possible).

    None of these are complete solutions however, so that's what we'd like to deliver later this year -- something that supports reuse of bits of build configuration across repositories.

  4. Danielo Rodriguez

    I'm also very interested on this. Currently our pipelines have too much repeated code, and having to change the setup involves changing all our pipelines, which is time consuming and error prone. We already npm packages that encapsulates the setup logic, but even for use the scripts there are some setup steps required that we are forced to repeat over and over again.

    All above workarounds involve some kind of duplication

  5. Terje Andersen

    +1 for interest. My team is considering the move from Jenkins, and we think Bitbucket pipelines seems promising. But to re-write steps that will be exactly similar across branch and tag pipelines seems like an hassle (not a complete show-stopper though, especially since you have some ideas on how to resolve this).

  6. Adam Copley

    Also considering the move from semaphoreCI. It would be good to be able to define "setup steps" that can be written once and run in any branch configuration's respective pipeline.

    Similarly, it would also be nice to have the option of whether or not to check out the repository again at the beginning of each build step to allow for nicer grouping of different tests without wasting build minutes on checking out again.

  7. Mina Luke

    I have the same issue. I am using different branches to utilize CI, staging, and production environments each of which has 2 steps and most of the scripts are the same so I want to share some of them.

  8. Jochen We

    hey guys. i had the same requirement. but since pipelines support yaml-aliases, you can use aliases to reuse steps. you can do somesthing like that:

    stepdefinitions:
      - builddeps: &builddeps
          name: install deps
          script:
            - dist/bitbucket/before_install.sh
            - dist/bitbucket/install.sh
          artifacts:
            - vendor/**
      - lintingtesting: &lintingtesting
          name: linting, testing
          script:
            - dist/bitbucket/lint.sh
            - dist/bitbucket/test.sh
    pipelines:
      branches:
        master:
          - step: *builddeps
          - step: *lintingtesting
      default:
        - step: *builddeps
        - step: *lintingtesting
    

    hope this helps.

  9. Mohammad Norouzi

    +1 This is really important as the steps are growing, the file becomes bulky and unreadable... plus it's easy to make unwanted mistakes because people are copying and pasting the shared steps

  10. Aneita Yang staff

    Hi everyone,

    Thanks for your interest in this feature.

    I'd love to speak to some users about their requirements and expectations of this feature. If you have time for an informal 30min chat, please send me an email at ayang@atlassian.com and we can find a time.

    Thanks,
    Aneita

  11. Mohammad Norouzi

    @Aneita Yang

    Sorry I'd love to talk about it but I don't really have time... however, to let you know what we really need, I've provided an example... something like this:

    image:
      name: dtr.abcd.com/xyz-devops/services:xyz-server-latest
      username: $DTR_USERNAME
      password: $DTR_PASSWORD
      email: $DTR_EMAIL
    
    pipelines:
      common:
        'step-common-one':
          script:
            - sed -i -e "s@license=@license=$WINDWARD_LICENSE@g" WindwardReports.properties
            - export MAVEN_OPTS=-Xmx2G && export M2_HOME=/opt/apache-maven-3.3.9 && export PATH=~/.local/bin:$PATH:/opt/apache-maven-3.3.9/bin
        'step-common-two':
          script:
            - aws s3 sync target/cucumber-html-reports/ s3://abcd-dev-cucumber-reports/digital-documents/develop --delete
            - rm WindwardReports.properties && touch WindwardReports.properties
            - echo "license=" > WindwardReports.properties 
            - docker login dtr.abcd.com --username $DTR_PUSH_USERNAME --password $DTR_PUSH_PASSWORD
            - docker build -t "dtr.abcd.com/xyz/some-name:develop" .
            - docker push "dtr.abcd.com/xyz/some-name:develop"
      branches:
        '*scratch*':
          - step:
              script:
                - echo "Skipping the build for a scratch branch"
        'develop':
        # can have multiple 'step' node
          - step: step-common-one 
          - step:
              caches:
                - maven
              script:
                - mvn -B deploy -Dmaven.repo.local=/root/.m2/repository -Djsse.enableSNIExtension=false -DaltDeploymentRepository=MyCentralRepo::default::https://nexus-portal.abcd.com/content/repositories/MVN-SNAPSHOT
              services:
                - mongo
          - step: step-common-two
          - step: step-common-three #Only to demonstrate that we may need more 'step' nodes  
    
    definitions:
      services:
        mongo:
          image: mongo
    
    options:
      docker: true
    

    So there are steps that can be defined in common section and can be used anywhere. I have removed a lot from our bitbucket file to make it simple

    instead of having multiple 'step' node we could also have 'before' and 'after' as long as they are all done in the same scope/session - meaning env variables and executions result are available across the three before/step/after phases

  12. Aneita Yang staff

    Thanks for the example @Mohammad Norouzi. Are your steps / common scripts only used within the single repository or do you see yourself having the need to use the same script across multiple repositories?

    For the former, you can achieve a similar result today by:

    • using YML references which lets you define the step once and reference the step from elsewhere in the same YAML file
    • putting the common step into a script in your repository which you can then call from your bitbucket-pipelines.yml file

    I'm interested to understand whether these options suit your needs and if not, why that is.

  13. Thomas Raehalme

    We basically have the same bitbucket-pipelines.yml in many repositories. Also many branches share the same basic part of the build and then add some branch specific functionality, eg. starting or finishing the release and deployment to test/prod.

  14. Mohammad Norouzi

    @Aneita Yang

    A lot of those steps are common across all other repositories and we have lots of them (almost 50 or more repos) so yes if they can be somehow shared, that'll be really useful.

    If we could define those steps as a common configuration that is available in all repos that's ideal. It also saves time when we have refactoring or modifications. For example, a month ago we needed to make some changes to support semantic versioning and I basically had to check out each and every repo and copy/paste the same steps whereas I could easily change that in one place and use them in each repo in the correct order. This is because as Thomas mentioned above the usage is both vertically and horizontally (multiple branch name as well as multiple repos)

    So yes, the YAML references may help in one repo but it really won't fix the issue. In my example above, I was just trying to say how we need to re-use those steps...

    So for example, the windward license you can see in one of the steps in my example, is shared across nearly 10 repo, however docker commands are used in all repos.

    Hope that helps

  15. Steven Alvarado

    I'll preface this by saying I do want a shared steps feature – possibly defined at the Team level similar to what you can do with Pipeline environment variables – but there is somewhat of a workaround. It's not ideal, but it works.

    What you can do is create a custom Docker image for a particular project type. Write the shared steps out as bash scripts (or your language of choice), then add them to the Dockerfile and build/push the image to use in your bitbucket-pipelines.yml config, for example:

    FROM php:7.1.1
    
    ## install and configure laravel dependencies
    ADD bootstrap.sh /usr/local/bin/bootstrap
    RUN chmod +x /usr/local/bin/bootstrap
    RUN bootstrap
    
    ## set up script for pipelines to deploy results to build server
    ADD deploy.sh /usr/local/bin/deploy
    RUN chmod +x /usr/local/bin/deploy
    

    When this is set up, you can use deploy in a step within your bitbucket-pipelines.yml file, like this:

    image: {your_organization}/{image_name}
    
    pipelines:
      default:
        - step:
            script:
              - ...
              - deploy
      branches:
        master:
          - step:
              script:
                - ...
                - deploy
    

    This way, if you need to update how deploy works, you can simply update the Docker image and all repos which use the image will be updated at once. It would be really nice to have this as a Bitbucket feature (i.e. scripts that can be defined at the Team level) rather than having to make a custom image, though.

  16. Mohammad Norouzi

    I think having named steps that are available in all repos is good idea... this way each repo can use any combination of steps.

    It can also become a team's/company's standard toward building/deploying their artefacts as opposed to each repo doing it its own way

  17. Radoslaw Janas

    I think we're missing one important thing in the discussion: passing parameters to steps! (it doesn't help much if [namedStep] is not aware of the context in which it runs).

  18. Steven Alvarado

    That's a good point, @Radoslaw Janas. If there was a way to have team/account-level scripts like I described above, you could pass params. As it is right now, you can use that custom Docker image method and accept params to your custom script when you call it in bitbucket-pipelines.yml, but again, it's kind of clunky and would be much nicer if this was an integrated Bitbucket feature.

    BTW you may already know this, but there are default environment variables which are automatically set that you can use to determine a decent amount of context when running scripts: https://confluence.atlassian.com/bitbucket/environment-variables-in-bitbucket-pipelines-794502608.html#Environmentvariables-Defaultvariables

  19. Raul Gomis staff

    Hi everyone,

    Thanks for all of the feedback on this issue. Some good news - we've started speccing out a feature that will let you share scripts between multiple repositories. This will help reduce the amount of repeated configuration across your repositories and make your bitbucket-pipelines.yml file more concise. We'd love to understand whether this is something that will suit the needs of your team:

    • You can share scripts across repositories by creating a task.
    • Tasks are built on Docker which gives you the flexibility to use any language or CLI to define your task, and also has the benefits of isolation (tasks won't affect the execution of another task) and reproducibility.
    • Tasks are defined in separate repositories which helps provide versioning capabilities.
    • To define a task, you will provide a YAML file with information about the task including:

      • name
      • description
      • base Docker image
      • parameters that are required by the script
      • script commands.
    • We will use Pipelines in the task repository to automatically build a Docker image for your task and push it to your specified Docker registry.

    • You can use tasks in your bitbucket-pipelines.yml file by providing account and repo name (account/repository) as the task ID, version number and the parameters required by the task (passed as environment variables).

    An example of how you might use a task looks like this (keep in mind that this is just an example and that the syntax has not yet been finalised):

    pipelines:
      branches:
        master:
          - step:
              name: Deploy to test
              deployment: test
              script:
                - echo "Starting deployment"
                - task: account/my-deploy-task:1.0
                  name: Deploy docker image to test environment
                  parameters:
                    IMAGE: my-image:latest
                    USERNAME: $SERVER_USER
                    PASSWORD: $SERVER_PASS
                - echo "Finish deployment"
    
    • This model also allows us to provide a set of supported tasks which will simplify the configuration for using Pipelines with other tools (e.g. AWS Lambda, AWS S3, Kubernetes, etc.).

    As mentioned, this solution allows you to define and share scripts between repositories. If you're interested in reusing configuration within the same bitbucket-pipelines.yml file, you can use YAML references to do this today. We'd love your feedback on our proposed solution and to understand whether it suits the use cases described on this issue - you can comment on this issue or send us an email at pipelines-feedback@atlassian.com with your thoughts.

    Thanks for helping us improve Pipelines!

  20. Cornelis Hoeflake

    @Raul Gomis This is great news! Personally I like this solution! It is clean and effective.

    Just to have some things on the whishlist :), allowing a shared bitbuket-pipelines.yml would be even more great. That allows us to use convention over configuration with minimal configuration per repository. For example:

    pipelines:
      shared:
        config: account/my-build-config:1.0
        parameters:
           TARGET: xyz
           DEPLOY: value
    

    But the given solution is a great step forward!

  21. Mohammad Norouzi

    @Raul Gomis Thanks. It seems good. I especially like the version of the task, it lets us to upgrade the scripts smoothly and without breaking all repos.

    Just wondering, what is name: Deploy docker image to test environment and is it mandatory? I think it's better to be optional to avoid too many lines.

    Also, do we need to pass the image as a parameter to the task? In other words, isn't that better some parameters like the image and also env variables defined in the current context to automatically be sent to the task without the need of declaring them? This makes bitbucket-pipelines.yml tidy and easy to read.

    My concern is, we end up have a huge list of parameters. For example consider 3 different tasks each of which has 5 to 10 parameters. Unless we come up with an idea to define context and perhaps those parameters that are marked as 'context-aware' can be automatically be transferred across all the tasks.

    Something like this:

    pipelines:
      branches:
        master:
          - step:
              name: Deploy to test
              deployment: test
              script:
                - echo "Starting deployment"
                - context: set USERNAME=abcd 
                - context: set PASSWORD=abcd 
                - context: set IMAGE=my-image:latest 
                - task: account/my-build-task:1.0         # all env variables defined as 'context' will be pass in to the tasks and are available here as well
                - task: account/my-semver-task:2.1     # all env variables defined as 'context' will be pass in to the tasks and are available here as well
                      parameters:
                          UPGRADE_TYPE:$MAJOR_CHANGE
                - task: account/my-deploy-task:1.0     # all env variables defined as 'context' will be pass in to the tasks and are available here as well
                - echo "Finish deployment"
    

    Also, I am in favor of @Cornelis Hoeflake 's idea

  22. Raul Gomis staff

    Hi,

    Thanks for the feedback! Exactly, the proposed task model will be a step forward to help you keep your bitbucket-pipelines.yml configuration files clean and maintainable across repositories. In the future, we could also consider reusing parts of bitbucket-pipelines.yml file or even the whole file. We'd like to keep getting feedback about all your specific reuse needs and concrete use cases / examples and validate wether the proposed tasks model is something our users might be interested in.

    @Mohammad Norouzi, regarding the specific details of the proposal (keep in mind that this is not yet finalised and things might change slightly):

    • Task name would be optional.
    • The base image of the task would be declared in the metadata file in other repo, not when using the task. The parameter IMAGE in the example was just an example of an task that deploys a docker image into a registry (nothing related to a task declaration).
    • Parameters would be passed as environment variables to the task script. In the task creation script users would be able to set mandatory parameters and default values, so that as little code as possible is required when using tasks in the bitbucket-pipelines.yml file. Setting mandatory parameters would make tasks more readable as parameters are explicitly declared. However, with this approach, parameter lists might be longer when using a task, especially if you use several tasks that require same parameters. That's not yet decided, but it's something we'll take into account.

    For example, task metadata file for Kubernetes might look like this:

    name: Deploy to Kubernetes
    description: This task deploys to Kubernetes
    baseImage: atlassian/pipelines-kubectl
    version: 1.0
    environment: # required fields only
      - name: APP_NAME
        default: "my app name"
      - name: CLUSTER_NAME
        default: "my cluster name"
      - name: KUBERNETES_HOST
      - name: KUBERNETES_USERNAME
      - name: KUBERNETES_PASSWORD
      - name: IMAGE
    script:
      - kubectl config set-cluster $CLUSTER_NAME --server=$KUBERNETES_HOST
      - kubectl config set-credentials deploy-user --username=$KUBERNETES_USERNAME --password=$KUBERNETES_PASSWORD
      - kubectl config set-context deployment --cluster=$CLUSTER_NAME --user=deploy-user
      - kubectl config use-context deployment
      - kubectl set image deployment/$APP_NAME $APP_NAME=$IMAGE:$BITBUCKET_BUILD_NUMBER
    

    and this is how it'd be used across your repositories:

    pipelines:
      branches:
        master:
          - step:
              name: Deploy to test
              deployment: test
              script:
                - echo "Starting deployment"
                - task: account/my-kube-task:1.0
                  parameters:
                    APP_NAME: my-app-name
                    CLUSTER_NAME: my-cluster-name
                    IMAGE: my-image:latest
                    KUBERNETES_HOST: my-kube-host
                    KUBERNETES_USERNAME: $KUBERNETES_USERNAME
                    KUBERNETES_PASSWORD: $KUBERNETES_PASSWORD
                - echo "Finish deployment"
    
  23. Aneita Yang staff

    Hi everyone,

    I've created two new tickets to track the main requests that we've identified on this issue. With this current issue, it has become difficult to understand who is interested in what, and which request is more important to our users - the two main asks that we've identified are slightly different projects for us.

    • Issue #17182 now tracks the request to have better pre-defined templates to help configure Pipelines to work with other tools (e.g. deployments to AWS).
    • Issue #17181 tracks the ability for users to define their own configuration which can then be reused across multiple repositories.
    • If you are interested in being able to reuse configuration within the same YML file, you can check out YML references and anchors.

    Please vote for and watch the new tickets linked above based on what you're interested in seeing. This will help us gauge the importance of each request. If you have a suggestion that's not covered by the two tickets above, please raise a new ticket so that we can assess and track it separately.

    Thanks for the help!

    Aneita

  24. Log in to comment