NOAA NCEI Complete 'pub' directory

Create issue
Issue #40 duplicate
Borislav Iordanov created an issue for a few days now. First I almost filled the drive on my own server and I had to stop it. Then kicked off a download on Sakari's server, but the connection is so slow I ended up getting a separate machine with a 4 terabyte volume on AWS. So now I'm trying it there. On the only a part of this is claimed. And on our bitbucket I noticed issues #5, #10 and #17 which are about the same organization (NOAA NCEI ).

I will report with comments on progress.

Comments (8)

  1. John Baez

    Thanks, I will change the status of this issue to "Resolved" and indicate on our wiki that you've downloaded it to your drive.

  2. Former user Account Deleted

    This should be far more than 620GB if it is a full mirror of /pub. /pub/has alone is ~12TB

  3. Former user Account Deleted

    Basing all of my info off of which should be the same stuff as on but over FTP.

    /pub/has/ftpusage.txt shows:

    Filesystem       1K-blocks        Used   Available Use% Mounted on
    /dev/simfs     76235669504 12081838080 64153831424  16% /ftp/pub/has

    Which indicated ~12TB used on /pub/has.

    In addition, running a recursive size summation (in FileZilla) of /pub/data grows to >1TB pretty quickly (The asos-* dirs alone are ~1.5TB).

  4. Sakari Albert Maaranen

    @John_Baez has limited our budget to longer term four server that is about 40 (or 44) terabytes total. Let's prioritize according to the email I sent earlier. Technically there is no reason why we couldn't do this, but it would hog most of our capacity, and with our approach we would need to split it to three servers.

    Let climate experts prioritize.

  5. Log in to comment