Docker builds started failing with "Container ID cannot be mapped to a host ID" error

Issue #17319 open
Alastair McFarlane
created an issue

Recently, our builds have started failing with the message:

Error processing tar file(exit status 1): Container ID 165586 cannot be mapped to a host ID

I can't figure out which part is failing, though from browsing other issues it appears to be some issue with user remapping. However, the only existing issues I can find that sound similar are to do with multistage builds. The build is failing some time after the apt-get autoremove in the following:

# Copy requirements in
COPY ./requirements.txt /app/
COPY requirements/ /app/requirements/

WORKDIR /app
# Install dependencies and requirements

RUN apt-get update && apt-get install -y pkg-config git build-essential python-dev \
&& pip install --upgrade pip \
&& pip install --upgrade setuptools \
&& pip install -r requirements/server.txt \
&& rm -rf requirements/dependencies \
&& apt-get remove -y build-essential git pkg-config \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*

# Copy the rest of the app
COPY . /app/

# Custom system config
COPY server/ /

Official response

Comments (24)

  1. Bradley Wogsland

    This is really frustrating that builds have started failing out of nowhere, and we're seeing the exact same Container ID 165586 error. The CircleCI sol'n did nothing for us either. And our failing images are only ~310MB.

  2. Alastair McFarlane reporter

    I looked at that first - unfortunately I haven't a multistage build, and my build is already running as root! I tried adding a chown for each of the files, but it didn't seem to help. My sense is that this feels like the same issue, but perhaps in a slightly different place, as the workaround doesn't appear to apply.

    I'd love to be wrong, but I did try!

  3. Olivier Lacroix

    I have contacted support, who rolled-back the introduction of user mapping for our account. builds are back! This is definitely not a fix, but at least allows to get some work done.

    I have also been advised that a guide will soon be published as well to deal with that situation. To be notified, you can "watch" this page.

  4. Joël Wijngaarde

    I had exactly the same error. When I changed the following line in the Dockerfile:

    RUN mv $PHP_INI_DIR/php.ini-production $PHP_INI_DIR/php.ini
    

    to

    RUN cp $PHP_INI_DIR/php.ini-production $PHP_INI_DIR/php.ini
    

    the error disappeared.

    Hope this helps in finding a solution / work around.

  5. Nathan Burrell staff

    As a part of the ongoing security hardening process in Bitbucket Pipelines, we decided to enable docker user namespace remap to eliminate potential security risks that might affect our users.

    This change affected a small percentage of users using docker in their builds. So, we wanted to share the different limitations that this change introduced, and the actions required to make your builds work with the security enhancement introduced.

    Issues caused by limitations of userns remap.

    As mentioned in the docker documentation linked above, enabling docker userns remap introduces the following limitations:

    The following standard Docker features are incompatible with running a Docker daemon with user namespaces enabled:

    • sharing PID or NET namespaces with the host (--pid=host or --network=host).

    • external (volume or storage) drivers which are unaware or incapable of using daemon user mappings.

    • Using the --privileged mode flag on docker run without also specifying --userns=host.

    We already disallowed on pipelines --pid=host and --privileged containers however --network=host is a new limitation.

    In the interim we have added all users who currently use this feature of docker (as indicated by our analytics) to an exclusion list.

    We will be making changes to docker to assist you in moving from using --network=host to a solution that will involve you using container links, port mapping and providing a similar feature to docker for OSX with a host that resolves to the host for the build.

    We will release further communications as to when these changes will be made and the exclusion list will be removed.

    Docker build, removing files doesn't remove them.

    Due to bugs in docker, when files were modified (removed/alterered) during a docker build via RUN/COPY commands the changes weren't being reflected in subsequent layers.

    We have upgraded docker to pull in the fix for these issues.

    https://github.com/moby/moby/issues/36648

    https://github.com/moby/moby/issues/35709

    https://github.com/moby/moby/issues/32919

    https://github.com/moby/moby/issues/30110

    However some failures require user intervention, as some commands were modifying the UID/GID (similar to the error below). For these situations docker now supports --chown as an argument to some dockerfile commands.

    Builds fails due to error: “Container ID Cannot Be Mapped to Host ID Error”

    Due to the way userns remap works (by remapping UID/GIDs to another less privileged UID/GID in the hosts namespace), UID/GIDs that are placed on files must be in the range 0-65535.

    If you recieve this error, you will need to perform a fix on the image that has this invalid UID/GID.

    If your recieved this error against a file that you created and placed into an image, the fix is to simply change the UID/GID of the file to one within the range.

    To find the file, first get the invalid UID/GID from the error message and run the following command on the files within the container.

    find / -uid|-gid <invalid-uid-or-guid> -ls
    

    To fix the ownership of the file use the following command

    chown -R UID_IN_RANGE:GID_IN_RANGE filename
    

    If you received this error against a file that was already present in a layer of the base image you depend on, you will need to raise a support case and work with the image maintainers of the base image to get this resolved.

    Test containers

    If you use the java library test containers there is currently a bug in that it doesn't run against a docker daemon with userns remap enabled. For more details and questions, please comment on this issue: https://github.com/testcontainers/testcontainers-java/issues/712

  6. Tomasz Tyzaj

    Same for me. I have this issue when running python2 pip on requirements file and when 'removing intermediate container'. I tried to change ownership, run as root. The image which I run does not have file with this ID. Weird thing.

  7. Nathan Burrell staff

    Hey just an update on this issue.

    As the pinned post above suggested we are removing the ability to be whitelisted to work around these issues (as we need to leave the change in place for security purposes).

    I have written this post however to assist you in fixing these issues for good in your builds (so they can continue to run in docker both with and without user namespace remapping).

    Changes to make your containers more secure on Bitbucket Pipelines.

    Kind Regards,

    Nathan Burrell

  8. Ashvin Narayanan

    Hey @Nathan Burrell , would help if you or your team actually read these messages. The "Container ID Cannot Be Mapped to Host ID Error” we are getting is not for a single file, as your pinned post suggests. And simply flagging it as a problem with the base image doesn't help any of us relying on python.

  9. yennie@lucasware.com

    This will probably get worse for python repos as those with pip cache optimizations cycle through caches: I am assuming that the cache feature is unaffected, yet when the cache is out of date or deleted, the image will then be subject to this change.

    The solution of finding the files in the image is null for pip installs. There appears to be a process where pip may have intermediate files in userns sensitive areas in the setup process that are then deleted. This makes it impossible to find the files dynamically.

    For a work-around (python pip installers with issues), try using the pip --target flag to control where this process takes place:

    RUN mkdir -p /src
    RUN pip install --no-cache-dir --target=/src -r requirements.txt
    RUN export PYTHONPATH=$PYTHONPATH:/src
    

    This way /src should already be user mappable, though you may have to chown of the /src directory.

    On another branch of this issue: https://github.com/moby/moby/issues/34645 has not been closed. Using multi-stage builds to copy files from docker images is now a broken feature in bitbucket pipelines without hacking around.

  10. Log in to comment