docker cache appears to need manual deletion to work consistently

Issue #16399 duplicate
Rachael Ludwick
created an issue

As we transition to pipelines, I've been trying to sort out our docker caching so that it reliably caches (and to fix some actual bugs that had crept in but weren't terribly noticeable with our previous build system). I got it caching but then as I kept tweaking the Dockerfile, I noticed it stopped caching. First I checked locally with docker on my mac (fairly recent version of docker) that:

  • if I built twice in a row, the second run found cache for all layers
  • if I built once, then modified some file in our source tree, the cache was invalidated at the correct step (the on that adds most of the source)

I then confirmed that if I rebuilt my branch with that dockerfile multiple times in pipelines that it wasn't using cache. It showed that it was downloading the cache during the "Build setup" phase but then at the end it didn't upload any new cache and the layers were all rebuilt.

I then deleted the Docker cache in the web UI:

Screen Shot 2018-06-01 at 10.53.55 AM.png

Then I ran the build twice again and it cached properly.

So it appears that the docker cache is very static -- that is, it isn't extracting all layers built by the current run of docker build and adding them to the existing cache. It only uploads any cache for future use if there was no cache to start.

Is this a correct description of how caching works? If so, is there any plan to fix this? That is not how docker caches when you run your own build server with its own docker daemon.

The major issue here is if a developer modifies the Dockerfile (even adding comments to a line can invalidate a step!) then the cache becomes useless, extending our build time. Not all developers are going to remember to go delete the cache manually when they merge such changes to master.

Comments (3)

  1. Fyodor Paligin

    I'm experiencing the same issue.

    I have a pipeline which consists of 4 steps where each step builds one image.

    Each step has definitions: services: - docker caches: - docker But the cache is updated only for the 1st step and only if it didn't exist before. In all other cases the image is built without the cache and at the end of each step I get this: Cache "docker": Skipping upload for existing cache

    In spite there were layers built.

  2. Aneita Yang

    Thanks for reaching out and for the suggestion. I'm going to close this issue as a duplicate of #16314 which is an open request to update existing caches when dependencies are updated. I'll add a note to the issue about the specific request from this issue about updating the docker cache when the Dockerfile is updated.

    @Fyodor Paligin, it sounds like your use case is slightly different, where you build multiple Docker images but only one docker cache is supported. If you're interested in being able to cache the layers of multiple Docker images, can you please create another request for us to track separately?

    Thanks,
    Aneita

  3. Log in to comment