Configurable timeout or hang detection in Pipelines to avoid wasting build minutes

Issue #13491 resolved
Richard Simko
created an issue

As a developer I would like to be able to set the Bitbucket Pipelines build timeout. We're using submodule which means that sometime people push commits containing references to SHAs in other projects that haven't yet been pushed, causing the build to hang. This causes 150 build minutes to be wasted when a normal build for us is 5-10 minutes. It would be great if I could set the timeout myself, thereby lowering it by a lot and catching these hung builds much earlier.

Comments (27)

  1. Toby Murray

    We've also seen this happen. Generally it has been because of a misconfiguration issue (e.g. PostgreSQL server not starting, error in Dockerfile) but it's an easy mistake to make and it burns through build time super fast if, for example, all builds using one faulty container hang.

  2. Joshua Tjhin staff

    Hey all,

    Conserving minutes from hung builds is something that we've thought about. Another idea we've toyed with is failing if there's been no log output for 10 mins. This would reduce the need to add configuration. Just wondering, of all your hung builds, did they have no logput as a result of being hung?

  3. Matt Jonker

    My last "hung" build was the result of an issue connecting to Github using npm install, looking like this:

    The authenticity of host 'github.com (192.30.253.112)' can't be established.
    RSA key fingerprint is ...
    Are you sure you want to continue connecting (yes/no)? |/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-...
    

    Looking at the documentation, the spinner seems to be disableable, but that produced output from the spinner until the build was cancelled after 2 hours.

    ```

  4. Matt Ryall staff

    We have a backend change rolling out in a couple of weeks that will make all our builds use non-interactive shells. That will take care of many of these issues where builds hang and use up build minutes.

    Leaving this issue open until we have a more complete solution to the problem.

  5. Frederick Zhang

    Hate to just jump in out of the blue but may I know whether there are any updates to this issue?

    Personally I think two parameters - max_build_time, max_freezing_time - would be sufficient. The first one is simply a limitation to the total time of a pipeline, the second one is the maximum time for a pipeline to run without generating any output. Both of them should be optional. Travis CI has some similar stuff.

    If the max_freezing_time is a bit complex at least I would like to see the max_build_time to be rolled out in the near future.

  6. Rob Dempsey

    After just having 4 or 5 builds go for two hours, causing us to go over our usage limit, this would be a very welcome feature. Our build times would normally be lucky to be 10 minutes at this stage, but something hung and caused the extraordinarily long build times.

    So a configurable hard limit would be perfect.

  7. nopuck4you

    Seriously folks, some of us paying users will move to other services if this continues - everyone has pipelines in the cloud now. By not rolling out a max timeout feature you're basically milking us on minutes due to connectivity or configuration issues. I just had a build run 2x for 120 minutes when I know it was hung after 10 minutes and should have failed the build then. I run deploy scripts that interact with the servers I deploy to. Sometimes issues happen and the build hangs. I can't spend 2 hours waiting to find out that a build is in a bad state.

    Please provide an update on when you expect a change to roll out. Last comment was from February about a fix to shells being non-interactive but that doesn't solve true build hang issues.

  8. Alexander Regueiro

    No update on this still? We just lost 120 build minutes thanks to a hanging build... This is super important!

    A configurable timeout would be the best way to handle this, I think. Both for total build time and for "no log output".

  9. Matt Ryall staff

    We've just released a new feature that addresses this request. You can now configure a per-step time limit using the max-time setting, either on a specific step or globally in your Pipelines YAML under options.

    The max-time value must be a whole number of minutes greater than zero and less than 120, with a default of 120. Here's an example:

    options:
      max-time: 10 # configure default 10 minute timeout
    
    pipelines:
      default:
        - step:  # this step uses the default timeout of 10m
            script:
              - ... 
        - step:  # this step will time out after 20 minutes
            max-time: 20
            script:
                - ...
    

    More information in our documentation here: bitbucket-pipelines.yml#ci_max-time.

    If you have other more detailed requests (like terminating due to lack of output), please raise a separate ticket for those. We believe this one addresses the main problem here, so we'll resolve this ticket.

  10. Log in to comment