Add scheduled builds to Pipelines

Issue #12737 resolved
Ben Tatham
created an issue

We need to run nightly builds, separate from commits in Pipelines.

We do this because we want to run a versions update to pick up our latest releases of internal libraries to ensure that nothing is broken. Since the dependencies are typically in a different git repo, we need to run this nightly (or on some schedule) to go check the artifact repository (nexus) for new releases of those libraries.

Official response

Comments (52)

  1. Mythili_Premkumar

    Yep I need to move code from Bitbucket to Salesforce on a button click as we ll as through schedules. we need to trigger the deployment on a button click. Please implement this feature too.

  2. Sten Pittet

    Hi,

    We're looking for a few customers that we could talk to regarding the implementation of scheduled pipelines. One the main question for us is to understand what is expected to run (special cron pipelines or a regular branch pipeline).

    If you're keen on having a conversation please reach out to me at spittet@atlassian.com.

    Thanks,

    Sten

  3. CarePay Limited

    We currently have several use-cases: 1) Nightly run of SonarQube (to measure code quality) 2) Nightly deployment to our test-servers 3) Deployment to QA servers (triggered by one of our testers, so would be nice to be able to configure permissions as well)

  4. Ben Tatham reporter
    1. nightly builds that builds against all the latest dependency versions automatically, so slightly modified from regular build.
    2. Potentially SonarQube run.
    3. Potentially running deeper (longer) integration tests.
    4. Push-button release builds (user initiated, not scheduled -- ideally from the Jira Release Hub, like Bamboo does today).

    So to answer the question -- different pipelines need to be run on the schedule.

  5. Sergey Parhomenko

    We did make most of the use cases above (SonarQube, longer builds, deployment) part of our pipeline, but also have once-a-sprint builds (that we currently have to trigger manually) to bump up the version number and to rebuild the Docker image used by Pipelines to make sure any fresh dependencies are cached. Generally, I would say the need for scheduled builds in our team comes up when we speak about one of these scenarios:

    • the need to avoid concurrent builds competing for a non-concurrent resource (SonarQube, deployments, FTP uploads);
    • the need to run expensive build steps less frequently to avoid wasting Pipelines minutes (integration tests, SonarQube, deployments);
    • the need to run time-sensitive builds steps in the specific time windows (non-rolling-update deployments in the pre-agreed time window, version bump up after the end of the sprint).

    There are other approaches to answer some of these needs, but scheduled builds do look like the simplest solution to all.

  6. Barry Lagerweij

    Integration with JIRA would be really nice, or even better: offer a REST API to trigger builds from a 3rd party system. Our QA users would be triggering the builds, but I would prefer not having to upgrade our plan for BitBucket because QA needs to trigger a build.

    If we have a REST API, it should not be hard to trigger the builds from cron or JIRA

    Regards, Barry

  7. Sten Pittet

    Thanks for the feedback. We're thinking of having a special entry in the YML for the definition of the scheduled pipelines. Could you give me your feedback on the following idea:

    • Schedules would be configured in the UI where you would select:
      • The branch you want to build.
      • The schedule you'd like to apply.
      • The pipeline you'd like to run.
    • The YML definition of the scheduled pipelines would be in a separate section to allow you to have multiple scheduled pipelines for a single branch.
    # This is a sample build configuration for Javascript.
    # Only use spaces to indent your .yml configuration.
    # -----
    # You can specify a custom docker image from Docker Hub as your build environment.
    image: node:4.6.0
    
    pipelines:
      default: # This is the default pipelines
        - step:
            script: # Modify the commands below to build your repository.
              - npm install
              - npm test
      scheduled: # In this section you define the pipelines that you want to be able to schedule.
        sonar:
          - step:
              script:
                - write your command here
        nightly-deployment:
          - step:
              script:
                - write your command here
      branches:  # This section contains the pipelines that need to run automatically on a push to a branch.
        staging:
          - step:
              script: # Modify the commands below to build your repository.
                - npm install
                - npm test
                - export HEROKU_APP_NAME=$STAGING_APP
                - ./heroku_deploy.sh
    

    In this case I could have the following configuration in the UI: Run nightly-deployment at 1AM every night from branch master.

    Looking forward to reading your feedback.

  8. Joachim Dorn

    Hello,

    that seems one option and would work for us. My 2 cents on a different approach:

    Currently the yaml-config defines only the "what", whereas the "when" is implicitly fixed (on each new checkin) But wouldnt it be more general and also more flexible to really expose the concept of a "trigger" in the syntax? People basically ask for other triggers. But there are no triggers e.g. for: "a cron expression fires" ; "a REST endpoint gets invoked" ; "some time has passed since last checkin" - I would try to make this "when" and "what" freely combinable.

    Im not good at yaml syntax, so I dont suggest an example, but my direction would be: 1) have a section "triggers" as sibling to "pipelines". There you define named triggers (e.g. time based: relative and absolut via "time passed" and cron-expressions). Implicitly there is always the current default trigger (new checkin) for backwards compatibility

    2) allow to reference one or multiple named triggers from the pipelines. If there is no named trigger, implicitly the default trigger applies.

    Cheers

  9. Ben Tatham reporter

    @Sten Pittet - looks like a reasonable idea to me. I couple of minor points, that I think can be easily reconcilable.

    • There is a bit of ambiguity between default and branches though, to me. default, I presume means on every commit. So if a commit matches one of the branches, does it also run the default, or just the branch pipeline? For most flexibility, I would say that default only runs if none of the branches match. That way we can do what we want explicitly on a branch.
    • For branch matching, you give the example of a branch named staging. Would that also accept regex/globbing too, like feature/*?
  10. Artie Brosius

    @Sten Pittet I think this is good, but I think you might as well go all the way to detaching the "when" from the "what". My idea is similar in concept, but different in implementation from Joachim's (if I'm understanding him correctly). I think the yaml file should define only the what; that is, which Docker image, which branch, and which commands to run.* But the "when" should be removed from the yaml configuration and instead configured via the UI. Through the UI you would select a "section" (think Ant targets) of the yaml file (the "what") and then configure one or more schedules for that section to run; initial choices would be (a) manually, (b) on commit, (c) on a specified schedule.

    Thanks

    *As an aside, it would be nice to be able to assign a different Docker image to each "what" section

  11. Michael Muller

    I would also like to be able to use a different docker image for my nightly builds.

    My incremental builds use a docker image that has a pre-populated local maven repository. I would like my nightly/weekly builds to start with an empty local repository.

  12. Nick Reilingh

    Wanted to throw out my weird example use case: There's a vendor that sometimes changes an XML file on the web, and when they do, I want to run an XSLT transform on it. I would set up a nightly pipeline to check if the file was changed and run the transform.

  13. Dominique van Roon

    If you could create a webhook to start a pipeline that would also help I believe. It does not seem like a lot of work but that's of course something for you to determine and allows others to build schedule services that work together with pipelines.

  14. Ben Bates

    We would use this feature to regularly retrieve and check-in changes to metadata for our Salesforce production instance. As a workaround, we have a small scheduled heroku app that re-runs the pipeline each hour via the bitbucket API - not ideal as it's an extra moving piece (and cost). Looking forward to this feature being implemented!

  15. Joshua Tjhin

    Hi all,

    Thank you for all the feedback submitted via the survey. Good news! Scheduled pipelines is now in Alpha. If you have already signed up for the Alpha, it should be immediately available. Otherwise, please sign up for the Alpha.

    To set up a scheduled pipeline, you must first define a custom pipeline.

    pipelines:
      custom:
        staging:
          - step:
              script:
                - echo "Scheduled builds in Pipelines are awesome!"
    

    Schedules are accessible from a button on the main Pipelines screen, and a pipeline can be configured to run hourly, daily or weekly.

    scheduled.png

    Read more on our docs.

    Over the next few weeks, we'll continue improving the feature so please send us any feedback you might have to pipelines-feedback@atlassian.com.

    Thanks!
    Joshua Bitbucket Pipelines PM

  16. Francois Germain

    Hi @xtjhin,

    We have started using scheduled pipelines on our side and it works great so far! Our only problem is that the way we are going to use this, we need to get the pipeline in motion ahead of the schedule from time to time. It also helps when actually debugging some pipeline steps to be able to force it...

    Thanks!

  17. Marcus Schumann

    One thing that immediately comes to mind is the ability to "Only run if changes since last run". E.g. it keeps track of which commit was latest when running and it compares it to last time it run.

  18. Francois Germain

    I agree with Marcus on that. For now, I built a poor man's work around to save our minutes. Here's something that will work for daily schedules. Add that at the top of your pipeline (given you use git):

    • gitStamp=$(git log -1 --format=%ct)
    • echo 'git stamp is:' ${gitStamp}
    • nowStamp=$(date +"%s")
    • echo 'now stamp is:' ${nowStamp}
    • if ((${nowStamp} - ${gitStamp} <= 86400)); then echo Pipeline will execute; else echo no new code && exit; f

    For weekly replace 86400 by 604800 and for hourly use 3600.

  19. Cristian Cocioban

    We are running the scheduled pipelines and it's working great. We didn't get though an email notification when the scheduled pipeline failed (we have the email notification enabled and it's working when the pipeline fails after a commit). Would be great to have it for scheduled pipelines too.

  20. Marc-Andre Roy

    @xtjhin Another idea that comes to my mind would be to display the scheduled builds in their own space. Right now, CI builds and Scheduled builds are displayed together and it can rapidly become an hassle to find jobs when you have a lot of CI and Scheduled build running. Here's my suggestion. Why not displaying CI build and Schedule builds in their own space? Having a tab for the CI build and another for the Scheduled builds would be great. This can also be a filter at top. I don't know your UX guidelines for bitbucket, I'm just making suggestions here based on what I would do :)

  21. Luke Swindale

    Just had a look at the status of my latest scheduled build, on my projects overview page. I noticed that it says "a day ago by" <avatar of some random dude>. Apparently the avatar resolves from "https://bitbucket.org/account/null/avatar/32"... who happens to be this guy: Nathan Wright. As best I can tell he hasn't worked for Atlassian although he does work at Aha!, which does integrate with Jira.... just a bit of trivia...

    BTW, the scheduled builds feature is working exactly as required. Thanks!

  22. Joshua Tjhin

    Great news! We made scheduled pipelines available to all users last week!

    Thanks for all the feedback that helped us deliver this feature. For all the additional feedback, as it's my last week at Atlassian, I've passed it onto the rest of the team so they can use it to improve Pipelines. I would also encourage everyone to create new issues. For running scheduled pipelines only if the branch has changed, I've created a more general feature request for an option to not trigger a build if a build has been run for the same commit/pipeline.

    Thanks again!
    Joshua Tjhin
    Bitbucket Pipelines PM

  23. Tam Le

    Can you please add an option to run pipeline daily during weekdays only (don't run on Saturday, Sunday) or the days (Mon, Tues...) to run the pipeline at time X ?

  24. Log in to comment