There have been more than a few data targets that are simply too large for Azimuth to hold on to. Soon, I will have over 300T's of free space available (500T total), pending arrival of hardware. Once it arrives and is setup, I'd like to hit the ground running with a list of larger targets that other efforts haven't been able to mirror in full, or especially as a collection of parts. This ticket can be a good starting point for these targets.
So far I have a few size estimates that will be no problem to fit:
ftp.coast.noaa.gov: Size Estimation Ongoing
Please suggest any other targets available to be copied wholesale, and if possible, a size estimate of the complete dataset. But I am also happy to start size calculations on my own infrastructure if given a target suggestion. I'm mainly interested in ftp & rsync targets that I can hit and mirror completely, http sites are a lot more messy and I frankly don't have much time available to curate a website scrape.