Pulling images takes looong time

I've seen tickets with similarities to this, but not really the same so I'm opening a new one.

We just started with bitbucket pipelines, and run our own Docker image, but no docker commands inside the steps. However, 85% of our build time is spent pulling those images in the Build Setup phase, from a private Docker Hub repo. This also means that parallelizing builds is just wasteful for us. I'd expect the first run to be slow, but not subsequent ones, at least within a short time period directly after.

The docker image is 1GB, which takes ~2 minutes for you to download. Is this really needed/expected? Why can't you cache it on your build agents and principally cut that time down to 0? We have turned on docker caching in the pipelines.yml, but from what I understand that only affects docker builds inside our steps, right?

Is there anything we can do to speed this up or is this how it's supposed to be?