obtain sizes for top-level directories in FTP for NOAA's NCEI

Issue #77 resolved
Jan Galkowski
created an issue
bkirkbri commented 40 minutes ago
I posted some subsets above. I agree it's best to break up what's left. I can claim some of them. Do you have sizes for top-level directories?


The NOAA NCEI FTP site is, by my estimation, about 26+ Tb in size. bkirkbri of ClimateMirror asks if I can provide sizes of top-level directories. I will do that as a task Tuesday, 17th January.

Comments (6)

  1. Jan Galkowski reporter

    I have results for these sizes. These were obtained using lftp via the du command, as has now been documented in our Best Practices Wiki page.
    noaa-ncdc-ncei-ftp-subdir-sizes-2017-01-17_170639.png Note this is probably a lower bound. I received a number of 500 error codes during the run of the du against these directories, and, so, there were files whose sizes were missed.

    I will update when I have the final.

  2. Log in to comment