Uploaded image for project: 'Bitbucket Cloud'
  1. Bitbucket Cloud
  2. BCLOUD-16489

Non-deterministic pipeline failures "python: Invalid argument"

    XMLWordPrintable

Details

    Description

      Bitbucket Pipelines repeatedly but non-deterministically fails on executing python commands from within an activated conda environment, although I'm not sure if conda itself has anything to do with the problem.

      The problem so far never occurred locally or when running the commands within the Docker container we use in Pipelines. When rerunning a failed pipeline it's successful in 90% of the cases, sometimes it takes a second rerun. The error occurs across different repositories, at different places in the pipeline, sometimes not happening for weeks and then suddenly happening several times a day.

      The errors always have the same format:

      #!shell
      
      bash: /opt/conda/envs/environment_name/bin/python: Invalid argument
      

      We already did a lot of debugging here, mostly assuming we would hit some type of resource limitation, but the error sometimes already appears during the linting step, which is the first command we execute after activating the previously created conda environment:

      #!shell
      
      bash: /opt/conda/envs/environment_name/bin/flake8: /opt/conda/envs/environment_name/bin/python: bad interpreter: Invalid argument
      

      During our integration tests we spawn new python processes which have separated conda environments. Here the chances of failure are even higher. Running the pipeline of a certain commit three times, lead to 6/3/4 tests fails respectively (out of 6 tests in total), on previous commits it also used to work for all six. Again: Running those tests within the Docker container locally works every time.

      Any help or information would be highly appreciated, thanks in advance!

      Attachments

        Activity

          People

            Unassigned Unassigned
            8bfa5b62-31ed-4015-9f77-555902a92218 Deleted Account (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: