Allow building multi-architecture Docker images (e.g. ARM images)

Issue #15317 open
f4b1en
created an issue

Please allow --privileged flag to build multiarch docker images. According to this article, it is possible with Github + Travis : http://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/

Register qemu-*-static for all supported processors except the current one docker run --rm --privileged multiarch/qemu-user-static:register

Currently, the following error is return when running the pipeline: + docker run --rm --privileged multiarch/qemu-user-static:register --reset docker: Error response from daemon: authorization denied by plugin pipelines: Command not supported. See 'docker run --help'.

Thanks

Official response

  • Matt Ryall staff

    Thanks for the suggestion, I've renamed this to be a bit broader.

    To move forward with this, we need a proof of concept for how this can work without using privileged Docker commands, which we unfortunately can't support on the shared Kubernetes cluster inside Pipelines.

    If someone can get a build working similar to @Ramchandar's example above (I'm unsure what it is trying to mount - maybe this can be switched off?), we can look at how we can improve support for this directly in Pipelines.

    Note that the Linux capabilities available to containers in Pipelines are those in the default set used by Docker, as referenced in their security documentation. (There used to be a full list there, but it now links to their source code for the canonical list.) So getting this working locally with Docker with the default capability set (and no privileged commands) would also be something we could work from.

Comments (12)

  1. Aneita Yang staff
    • changed status to open

    Thanks for raising this.

    The privileged flag means that Docker will allow access to all other builds on the machine. For security reasons, we currently don't support this. We will need to do additional investigation to determine whether this is something that Pipelines will support in future. However, the team are currently working on other higher priority features, so this isn't something that we'll be working on anytime soon.

    In the meantime, I'll open this issue to gauge the interest of other users on this functionality.

    Thanks,
    Aneita

  2. Ryan

    This is a highly desired feature for us. We can avoid the --privileged command by running the QEMU files directly.

    I've attempted to do that below but am given an error when one of the scripts calls mount:

    bitbucket-pipelines.yml:

    # enable Docker for all steps
    options:
      docker: true
    
    pipelines:
      custom: # Pipelines that are triggered manually
        deploy:
          - step:
              script:
                - docker version
                # QEMU setup (for cross platform compilation)
                # unsupported --> docker run --rm --privileged multiarch/qemu-user-static:register
                - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/register.sh
                - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/qemu-binfmt-conf.sh
                - chmod +x register.sh qemu-binfmt-conf.sh
                - ./register.sh
                # Build for each architecture
                - docker build arm64/ -t image-arm64
                - docker build amd64/ -t image-amd64
                # Push to registry
                - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
                - docker push image-arm64
                - docker push image-amd64
    

    Build fails with:

    + ./register.sh
    mount: permission denied
    ./register.sh: 31: exec: /qemu-binfmt-conf.sh: not found
    
  3. Matt Ryall staff

    Thanks for the suggestion, I've renamed this to be a bit broader.

    To move forward with this, we need a proof of concept for how this can work without using privileged Docker commands, which we unfortunately can't support on the shared Kubernetes cluster inside Pipelines.

    If someone can get a build working similar to @Ramchandar's example above (I'm unsure what it is trying to mount - maybe this can be switched off?), we can look at how we can improve support for this directly in Pipelines.

    Note that the Linux capabilities available to containers in Pipelines are those in the default set used by Docker, as referenced in their security documentation. (There used to be a full list there, but it now links to their source code for the canonical list.) So getting this working locally with Docker with the default capability set (and no privileged commands) would also be something we could work from.

  4. Ryan

    Just an idea: @Matt Ryall could the Kubernetes cluster expand to arm64v8 (and even other) architectures? This way we can add a step to our pipelines YAML to specify the underlying bare-metal architecture.

    This way we wouldn't need to use QEMU to build our arm64 images, since it's already an arm64 environment.

  5. Matt Ryall staff

    @Ryan - good question. We currently run on Amazon EC2, which only runs Intel hosts. A switch to another hosting provider is doable, but is not something we're willing to consider right now.

    So QEMU or another similar emulation tool seems like the best path forward for building or running these images on Pipelines in the near future.

  6. reijosirila

    Our engineer where able to build ARM images using Modified version of qemu: https://github.com/balena-io/qemu

    There you have a additional QEMU_EXECVE flag , so with that you can use in pipeline:

    docker build -t abc -f Dockerfile.qemu .
    

    and Dockerfile.qemu includes something like....

    FROM arm32v7 ... 
    ...
    COPY qemu-arm-static /usr/bin/
    SHELL ["/usr/bin/qemu-arm-static", "-execve", "/bin/sh",  "-c"]
    RUN build-for-arm-script
    
  7. Log in to comment