Details
-
Bug
-
Resolution: Support Request
-
Medium
Description
Bitbucket Pipelines repeatedly but non-deterministically fails on executing python commands from within an activated conda environment, although I'm not sure if conda itself has anything to do with the problem.
The problem so far never occurred locally or when running the commands within the Docker container we use in Pipelines. When rerunning a failed pipeline it's successful in 90% of the cases, sometimes it takes a second rerun. The error occurs across different repositories, at different places in the pipeline, sometimes not happening for weeks and then suddenly happening several times a day.
The errors always have the same format:
#!shell bash: /opt/conda/envs/environment_name/bin/python: Invalid argument
We already did a lot of debugging here, mostly assuming we would hit some type of resource limitation, but the error sometimes already appears during the linting step, which is the first command we execute after activating the previously created conda environment:
#!shell bash: /opt/conda/envs/environment_name/bin/flake8: /opt/conda/envs/environment_name/bin/python: bad interpreter: Invalid argument
During our integration tests we spawn new python processes which have separated conda environments. Here the chances of failure are even higher. Running the pipeline of a certain commit three times, lead to 6/3/4 tests fails respectively (out of 6 tests in total), on previous commits it also used to work for all six. Again: Running those tests within the Docker container locally works every time.
Any help or information would be highly appreciated, thanks in advance!