Singularity: Pull image anew from shub when changed?

Create issue
Issue #1019 new
Christian Arnold created an issue

I noticed that when one uses Snakemake with --singularity-prefix, and the corresponding Singularity image is already present, Snakemake uses this one but does not check whtether the image is actually "current". In my case, I updated the Singularity recipe, it was rebuild on SingularityHub but Snakemake did not detect this and nevertheless continued using the "old" image in the prefix folder. I am not sure whether this is actually easily possible, but it should be checked and mentioned.

Comments (4)

  1. Christian Arnold reporter

    Ping, I think this should be addressed, manually deleting stored images from the singularity-prefix directory because the image has actually changed is really not a solution and error-prone.

  2. Johannes Köster

    The idea was that you define the image with a commit id or a tag. Otherwise, you will never get a reproducible result, because people would get the latest image instead of the version you used. But I agree that it should be communicated in the docs, and Snakemake should issue a warning if an image is defined without at tag.

  3. Christian Arnold reporter

    I see what you mean, but sometimes, I want them to use the latest version, and because I do not specify a particular commit, this should be detected, right? If the image file already exists on disk but has a different hash as compared to the image that is referenced via shub, don't you think this should be detected and updated? With the versions yes, then one specified versions in the Snakefile, and changing the version will force redownloading the image I suppose. But just giving an URL such as "shub://chrarnold/Singularity_images:difftf_conda" should be checked I'd say. Right now, I am manually deleting images from the prefix directory whenever I update my image, as this is still a pipeline in development.

  4. Arnar Flatberg

    Ping. This is something I got myself into as well. Typically during the development of snakemake workflows with containers. Would love to see that snakemake discovered if you have an updated container build (with the same tag or no tag). I imagine a typical workflow would be to use no tag or latest tag during development and then add a specific tag when you are ready for prod. I would argue that reproducibility is the developer's responsibility (you know .... don't change container with a specific tag when used in prod )

  5. Log in to comment