Issue #40 duplicate
Borislav Iordanov
created an issue

https://www1.ncdc.noaa.gov/pub/ for a few days now. First I almost filled the drive on my own server and I had to stop it. Then kicked off a download on Sakari's server, but the connection is so slow I ended up getting a separate machine with a 4 terabyte volume on AWS. So now I'm trying it there. On the climatemirror.org only a part of this is claimed. And on our bitbucket I noticed issues #5, #10 and #17 which are about the same organization (NOAA NCEI ).

I will report with comments on progress.

Comments (14)

  1. Jan Galkowski
    • marked as bug

    Nick Gregory reports:

    Nick Gregory commented on issue #40:
    NOAA NCEI Complete 'pub' directory
    This should be far more than 620GB if it is a full mirror of /pub. /pub/has alone is ~12TB
    
  2. Anonymous

    Basing all of my info off of ftp.ncdc.noaa.gov/pub which should be the same stuff as on www1.ncdc.noaa.gov but over FTP.

    /pub/has/ftpusage.txt shows:

    Filesystem       1K-blocks        Used   Available Use% Mounted on
    /dev/simfs     76235669504 12081838080 64153831424  16% /ftp/pub/has
    

    Which indicated ~12TB used on /pub/has.

    In addition, running a recursive size summation (in FileZilla) of /pub/data grows to >1TB pretty quickly (The asos-* dirs alone are ~1.5TB).

  3. Jan Galkowski

    Actually, Nick, you are correct, in spirit, if not in detail. I just did a directory size calculation on /pub. It took a while. I got 29.6 Tb.

    That's much for the heads up! I think we can do a part, but not all.

  4. Sakari Maaranen

    @John Baez has limited our budget to longer term four server that is about 40 (or 44) terabytes total. Let's prioritize according to the email I sent earlier. Technically there is no reason why we couldn't do this, but it would hog most of our capacity, and with our approach we would need to split it to three servers.

    Let climate experts prioritize.

  5. Log in to comment