Ability to run multiple docker containers in the same build

Issue #12757 resolved
Baris Tuncay
created an issue

We need the ability to use something like Docker Compose to define multiple images in the same build.

Currently, a user needs to create a custom container if they need a few different services (e.g. database + web application) running to run the whole testing suite

Official response

Comments (49)

  1. Chris Knight

    We would also love this. We link container instances for mysql, redis and elasticsearch to our application docker instance. Our application docker instance would be too big and bulky if we had to put all of these services into it and it doesn't strike me as being the proper way to use docker if we did that either.

  2. Chris Abernethy

    Would love to see this as well, our use case is wanting to confine specific tools to their own docker image and run multiple steps in a build, each using the appropriate image.

  3. Olivier Sallou

    +1, creating a container with database, messag services, etc. and whatever service is needed to test application is the contrary of microservices, and force to create many images to test against different database versions (different versions of mongodb or mysql for example). Starting a db image + application image + ... would help and ease the setup and testing.

  4. Sten Pittet

    Hi everyone,

    We're still working on having Docker build and push commands supported but as a workaround I've created a Docker image with Node.js and MongoDB that you can use with Pipelines.

    I have other images ready for Postgres and MySQL and the equivalent for Ruby but I first wanted to get feedback here before I publish the repositories and the documentation for the other example. Please have a look and tell me what you think.

    Thanks,

    Sten Pittet Pipelines Product Manager

  5. Alex Soto

    Looks like a good start.

    I think move your pipelines build steps into a single ./bin/build script and show how to simulate the pipelines build process by running the build script locally so it's more clear to someone not familiar with docker and pipelines

    Ex:

    docker run -i -t <your-docker-account>/node-mongodb /bin/build

  6. Toby Murray

    @Sten Pittet - the Dockerfile linked in your comment is not following a bunch of fundamental best practices, so far as I can tell (just learning the tooling, so may be off base). As a current Pipelines user, I think focusing work on Docker integration is far more valuable than providing containers. The Docker community at large is blowing up with containers, so why not find or adopt the various high quality containers that exist?

    Additionally investment in multi-service containers, or whatever you want to call them, seems like a bit of a waste as it seems to be contrary to the design Docker is pushing for. I strongly suggest reviewing Docker's own best practices document if you haven't looked at it before.

    Feedback:

    If you run docker history spittet/node-mongodb you'll see the number of layers you're creating on top of the base:

    IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
    72d50921d3a0        12 hours ago        /bin/sh -c apt-get install -y mongodb-org       211.5 MB            
    <missing>           12 hours ago        /bin/sh -c apt-get update                       466.4 kB            
    <missing>           12 hours ago        /bin/sh -c echo "deb http://repo.mongodb.org/   67 B                
    <missing>           12 hours ago        /bin/sh -c apt-key adv --keyserver hkp://keys   1.166 kB            
    <missing>           12 hours ago        /bin/sh -c apt-get update                       453.9 kB
    

    Is this intentional? Generally speaking, appending consecutive RUN commands with \ && works better as it will yield a single layer, e.g.:

    ...
    && apt-get update \
    && apt-get install -y mongodb-org --no-install-recommends \
    

    Minor, but cleaning up after oneself is also seen as a good practice, so a refactored Dockerfile might looks something like:

    FROM node:4.6
    
    RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 \
      && echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | tee /etc/apt/sources.list.d/mongodb-org-3.2.list \
      && apt-get update \
      && apt-get install -y mongodb-org --no-install-recommends \
      && apt-get clean \
      && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
    

    Which yields a single layer:

    IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
    215967ffd781        4 minutes ago       /bin/sh -c apt-key adv --keyserver hkp://keys   211.5 MB 
    

    Additionally, Docker containers aren't really OSs, so I'm suspicious that the container isn't working the way you're expecting it to. The team over at Phusion touches on this a bit in this article. Some of it is a bit controversial, but the take-away is that most of the time it doesn't work to just apt-get install <thing I want>. Taking the official Mongo Dockerfile (here) there are quite a few differences, but beyond that - why not just use their official Dockerfile for your own container? You could mash the Node and Mongo files together easily enough.

    Aside from that, your bitbucket-pipelines.yml is a bit concerning with respect to the assumptions it's making:

            script: # Modify the commands below to build your repository.
              - service mongod start
              - npm install mongoose
              - node test.js
    

    Typically the entrypoint will handle starting whatever the container is responsible for. I think it likely overlaps with #12864 but if the Docker containers need a bunch of Pipelines interfacing magic, it's damaging to the reusabilty and adoption of containers. As much as possible, it's nice if the logic can live in the container instead of in a script that interacts with the container. Obviously it's not easy to make everything work together seamlessly, but the more atypical things Pipelines needs, the less benefit there is to be had from the Docker community at large.

  7. Alex Soto

    I'm no docker expert, so maybe there is a work around, but the container can run only one process at start (in this case bash), hence the need to start mongod

    I've run into the same problem while building my own docker container for pipelines which have postgresql, since the container is launched and the pipelines steps are run inside bash, the container's process won't have postgresql auto-started

  8. Toby Murray

    but the container can run only one process at start

    Agreed - this is conventionally encapsulated in CMD or ENTRYPOINT. Generally speaking if /bin/bash is executed instead, then the first (or last) thing that would need to be done is run the entrypoint script, otherwise the container isn't running whatever it's designed to run.

    Yours is a prime example - as currently implemented, the PostgreSQL container will run PostgreSQL's docker-entrypoint.sh script when the container is started, which does a whole bunch of stuff to start the server. As soon as you use /bin/bash as the entrypoint, that initialization script doesn't get run and you'll need to run it yourself. That's a manifestation of not adhering to the "one process per container" idea. As it currently stands, our container (which also uses PostgreSQL) has the two following steps to get PostgreSQL started:

    - nohup /docker-entrypoint.sh postgres &
    - until PGPASSWORD=password psql -P pager=off -h "$host" -U "postgres" -c '\l'; do >&2 echo "Postgres is unavailable - sleeping"; sleep 5; done
    

    Until this ticket is complete, I'm not really sure of a more preferable way to invoke the build. Maybe it'd be nice to provide the option of either running /bin/bash or using the defined entrypoint script? That way we could create our own scripts that live inside the containers, and Pipelines could just run the container as is. The benefit there being that it would move more of the logic from Pipelines to Docker, where it's more encapsulated.

    For a concrete example, why does

    script:
      - npm install mongoose     
      - service mongod start     # Run this command to start the Mongo daemon
      - node test.js             # Replace this with any command you need.
    

    live in the Pipelines yml instead of in the Docker container's entrypoint script (beyond "because that's the way Pipelines works")?

  9. Alex Soto

    I would definitely not want to split and maintain special scripts that are project specific inside a container.

    What I do is have pipelines specific build scripts that are checked into my project which are invoked by the yml file

    ex:

    pipelines:
      branches:
        master:
          - step:
              script: 
                - ./bin/pipelines-build -e DOTENV.ci
    
  10. Sten Pittet

    Hi Toby

    Thanks for your feedback. My main goal was to create an example that would be easier for people that are not familiar with Docker to understand and reuse. I've looked into creating better optimized and much smaller images but then the Dockerfile becomes quite complex to read. That being said I can definitely make the current example better and you have great insights (as a Product Manager I'm still learning a lot about Docker).

    I'll keep iterating on it but our main goal is still to support running multiple containers in a single pipeline - this will be the better solution for it.

    Cheers,

    Sten

  11. Toby Murray

    our main goal is still to support running multiple containers in a single pipeline

    Awesome to hear that that's still the official plan, @Sten Pittet.

    I would definitely not want to split and maintain special scripts that are project specific inside a container.

    Right now each of our containers is pretty specific to the product it builds, so from my perspective the options seem to be "change the Docker container, then go and change the Pipelines yml" or alternatively just "change the Docker container". Maybe once our process matures it'll become more apparent what the best solution is.

    What I do is have pipelines specific build scripts that are checked into my project which are invoked by the yml file

    That's a good idea! We have configuration files and similar things that are added into the repo, but I didn't think of putting the entrypoint script itself in the repo. I think I'll investigate that a little more. Maybe even a two stage initialization, so the container can be pulled and run as is on developer's machines to set up the build environment, then the script in the repo actually executes the build.

  12. Andrew Nemzek

    +1

    Would love to use pipelines to drive build/test/deploy, but have a multi-container application. Not interested in creating a separate environment to run tests because it could introduce environment discrepancies. One of the larger reasons to use docker is to keep your environments (ie: dev/qa/prod) consistent.

  13. Joe Mewes

    +1

    find it odd that multi container builds isn't a default since so many application rely on multiple services and Docker's micro-service infrastructure is base don having a container per service. I think without docker-composer or at least multi-container build support we won't be able to adopt pipelines. is there an ETA for multi-container support?

  14. Chris Curwood

    I agree! We have multiple CI Pipelines that leverage different versions of Applications throughout the lifetime of our product and things like testing, sandboxing and prototyping, it can be necessary to massage and play with multiple containers.

  15. Matt Ryall staff

    I'm happy to let you know that we've started planning this feature in the Pipelines dev team. At this stage, I'd like to get some feedback from interested customers on our plans.

    If you are available in the next two weeks for a video call to discuss your build configuration and how you'd like to use multiple services/Docker containers, please send me an email at mryall@atlassian.com with a short summary of what you use Pipelines for. Given the high amount of interest in this feature, I'm hoping to find a handful of typical use cases to validate our plans against. Thanks!

  16. Chris Curwood

    Sorry Matt, couldn't wait. Really need Docker and our CI pipeline so we already bounced. This would have been better as an MVP candidate since this is one of the strengths of Docker. I may be willing to come back later but already invested in our current solution. Bitbucket is still my repository though.

  17. Alex Soto

    We've also moved on. This was reported last May! Given than the EOL of Bamboo Cloud is just days away we wished we could have migrated to pipelines, but we didn't have much choice.

  18. Steve Dumais

    Hi Matt,

    I work at TeachingStrategies, a youth education company in Bethesda, MD. We’re currently using pipelines to run unit tests for our front-end Javascript code, and unit/handler/api tests for our back-end Golang service code. For the Golang code, we’re currently cobbling a few pieces of technology together – simplest being Golang+Mongo – so that we can run the service and it’s backing data store to perform some proper API tests every commit. We’ve been really happy with pipelines thus far. It would make it better however if we could eliminate the need to have multiple things running in a home-grown container. Use a separate pre-built Mongo container for instance. I’d be happy to talk through this further if it’ll help push this process along.

    Thanks for the effort on this!

    Steve Dumais Manager of Technology Development Teaching Strategies, LLC

  19. johan dindaine

    I moved onto something else but what I wanted was to be able to run my docker compose setup or be able run multiple container the same way compose does. The simplest thing would be to be able to run compose.

  20. Andrew Nemzek

    I'm still waiting for BB to implement a multiple container scheme for pipelines before I start using it. I would still like to use it! I agree that being able to supply a compose file or something like it to specify the system to test against would be best.

  21. Derek Rosenzweig

    Adding my $0.02 here - I'm currently investigating use of Pipelines, and my org does not yet use Docker in any kind of production environment. Everything I've read about the subject, and everything we've prototyped while looking into it, has pointed towards multi-container environments. For instance, we need to build assets with NPM/node, then run a Jekyll build command which requires Ruby, then deploy using AWS. I should be able to do that without having to create a single custom image from scratch. Docker compose would be a great asset here.

  22. Andrea Sprega

    +1 for the feature. I am currently running pipelines on a self-hosted Jenkins, but it would be great if BB pipelines supported docker-compose out of the box (much like TravisCI with "docker" service enabled). I think the most generic thing you could do is provide an image with a running docker daemon and basically allow to use a docker-compose.yml file that also uses private images.

  23. Marco Massenzio

    Having to build a "custom" docker with all services crammed into it, just so I can run pipelines, is not only contrary to any best practice (not to mention: a major waste of our time) but also completely pointless: I'd be testing a configuration that we'll NEVER use anywhere (in dev, in QA, let alone in Production).

    What's the point?

    Until you enable the ability to run multi-containers (either with compose, or just via multiple invocation of docker run) this is really of little to no use for serious commercial development (or, even, for people like me who take seriously their code, even when doing "our own thing").

  24. Marco Massenzio

    Just noticed that this request (with currently 138 votes) has been open since May last year. I must assume that BB is not taking Pipelines seriously enough.

    Good to know, we won't use them. Nice try, sorry it didn't work.

  25. Matt Ryall staff

    Thanks for the feedback, Marco. We've been working on this feature for a few weeks now, and hope to have an early release available for alpha testers within a week or two. (Unfortunately, Bitbucket Issues doesn't let us mark the issue as in-progress.)

    If you're interested in trying it out, please email me at mryall@atlassian.com with your repo details, and I'll let you know as soon as it's available to try. I agree that it's a major missing feature, and we're working on it alongside Docker build and push as our top priorities in the Pipelines team.

  26. Andrea Sprega

    @Matt Ryall can you confirm your intention is to NOT support docker compose? If I understand correctly, I would have to re-wire my container by hand since bitbucket pipelines still do not support docker-compose.yml. And what if I need more than 3 containers? In my case, I would actually need 5 (small on average, definitely not demanding 1 GB of memory). I don't understand why every every container will be given a fixed share of 1GB memory. It would be much simpler if the memory limit was global and more freedom was given to the pipeline job.

    All in all, I understand the huge technical challenges, but honestly these pipelines are still far too limited to even be considered over a custom Jenkins setup (or pretty much any other competitor that supports docker).

  27. JoshuaT staff

    Hi @Andrea Sprega,

    We would still like to support docker compose, in a similar way to docker build and push support #12790 but first we need to work on enabling docker run #14145.

    The memory constraint is purely a technical one atm. Our current infrastructure based on Kubernetes doesn't allow us to share resources..yet. A memory limit that is global is indeed the ideal design. Please know that this is only the first version and we will improve it.

    Regards,
    Joshua
    Bitbucket Pipelines PM

  28. Matt Ryall staff

    We're happy to announce that we've made multiple containers available in Pipelines via our new service containers feature:

    https://confluence.atlassian.com/bitbucket/service-containers-for-bitbucket-pipelines-874786688.html

    We believe this feature will address the testing needs for the vast majority of our users. If you have a use case that doesn't fit within the limitations noted on the page, please raise a new ticket with details on your use case.

  29. Matt Ryall staff

    Just following up here because there were a few questions above about Docker compose support. I'm happy to share that with the support we added for docker run in December (#14145), it's now possible to run compose files directly in Pipelines.

    In the last two weeks, we also added configurable memory limits for services (#14752), which allows you to increase the aggregate memory available to your Docker containers (including those run via docker compose) up to 3 GB in normal mode, or 7 GB in large build mode (#13874, size: 2x). As the feature name suggests, you can also increase or decrease the memory allocation of individual service containers.

    Hopefully this covers off what everyone needs to run whatever additional containers you require for your Pipelines builds, but if not, we're happy to review any further requests as new tickets. Happy building!

  30. Log in to comment