pipelines run commands in kubernetes pods instead of our docker image

Issue #13811 resolved
Fabian Off created an issue

This just happened to several of our builds - instead of launching in our docker image, the build process was started in the parent kubernetes pod:

+ umask 000
+ git clone --branch="develop" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/simpletechs/some-repo.git $BUILD_DIR ; git reset --hard some-hash ; git remote set-url origin https://x-token-auth:{access_token}@bitbucket.org/simpletechs/some-repo.git
Cloning into '/opt/atlassian/pipelines/agent/build'...
HEAD is now at some-hash some-text
+ chmod 777 $BUILD_DIR
+ mkdir tmp
+ some_command
/opt/atlassian/pipelines/agent/tmp/bashScript39....sh: line 16: some_command: command not found

We were able to verify by running a couple of commands that this is, in fact, not our docker image:

# cat /proc/version
Linux version 4.7.3-coreos-r2 (jenkins@worker-1) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.5, pie-0.6.4) ) #1 SMP Thu Feb 2 02:26:10 UTC 2017
# ps auxw
    1 root       0:00 /bin/sh -c exit $(mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && cat /opt/atlassian/pipelines/agent/tmp/build_result)
    7 root       0:00 cat /opt/atlassian/pipelines/agent/tmp/build_result
    9 root       0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperShellScript2853231412848204416.sh
   15 root       0:00 /bin/bash /opt/atlassian/pipelines/agent/tmp/bashScript4045879203974059494.sh
   16 root       0:00 bash -i
   27 root       0:00 ps auxw

This is obviously a rather big deal, as our build require tools that are not present in the parent pod (and why should they be).

Comments (5)

  1. Samuel Tannous staff

    Hi Fabian

    When running in a Docker container, /proc/version returns the version of the host of the container so this is not evidence of your script running outside your build container. The listed processes are also consistent with everything we run inside your Docker image.

    We have made a change to our build infrastructure recently so you may have encountered a change in behaviour. This is unexpected and we'd like to rectify it ASAP. Could I ask that you please open a support ticket via https://bitbucket.org/support so that we can inspect your repository and work out what might have changed?

    Thanks Sam

  2. Fabian Off reporter

    Hi Samuel,

    thanks for your feedback. upon further investigation we can actually verify that we are running in our own image. The missing docker run and misleading /proc/version did confuse me, sorry.

    Please note though that our builds failed because making the shell non-interactive (as a result of the infrastructure change) also lead to aliases not being expanded. We were able to fix this by using this in our scripts:

    # fix aliases not being expanded because of non-interactive shell
    shopt -s expand_aliases

    Finally, can you please verify that your support staff actually got themselves access (without our consent) and that this is why the following showed up in our logs? The timestamps pretty much match the creation of this ticket. screenshot

  3. Kaleb Elwert Account Deactivated

    Hi @0ff,

    Could you open a support ticket about the repository access? We can definitely help clarify what happened there, but we generally try to avoid commenting on private information (like your repository logs) on this issue tracker, as it's completely public.

    Support can be reached by emailing support@bitbucket.org or via support.atlassian.com.

  4. Log in to comment