NOAA NCEI Satellite Datasets Web site

Issue #17 resolved
Jan Galkowski
created an issue


to Sakari's Datarefuge FTP server in


wget --dns-timeout=10 --connect-timeout=20 --read-timeout=120 --wait=12 --random-wait --prefer-family=IPv4  --tries=40 --timestamping=on --recursive --level=8 --no-remove-listing  -nv --output-file=ncdc-noaa-satellite-access-datasets-gov.log --follow-ftp --no-check-certificate

Comments (20)

  1. Jan Galkowski reporter

    This is being copied from the site you specify, still in progress. Plan as usual is to download it, and then transfer to Sakari's FTP server. This is going to my NAS. It currently has 57 GB in 9900+ files.

  2. Jan Galkowski reporter

    @marsroverdriver I do not know. In fact, we do not know. We know little of the semantics of these files, and, moreover, the master list of files was a survey and no attempt was made to identify duplication. Moreover, we've been told that even the master list at is both incomplete and redundant.

    Accordingly, given the situation, My philosophy on the matter is to simply replicate, since we can't discriminate. I'd rather have a couple of copies than no copies. Moreover, that's how the overall project is trying to constrain errors, although I don't know what people would do with file naming problems except, perhaps, to use different tools.

  3. Jan Galkowski reporter

    Had to suspend this one for a bit because needed to move the target to a different disk, since the previous hosting disk was running out of room. Will restart once at the new disk.

  4. Jan Galkowski reporter

    Ach! Going way too slow to the offline disk. Not making progress at all. Will restart on the server directly, hoping to make up time. Would have had to transfer by FTP from local disk to the server anyway.

    Destination is now:

  5. Greg Kochanski

    Set up a key on pub01 via ssh-keygen -b 3072 -t rsa, copied the public key to the bitbucket site, downloaded a copy of the repository to my account via git clone then ran bash -x azimuth-inventory/miscellaneous/ /var/local/pub/ to set pub01:/var/local/pub/ncdc..... to read-only.

  6. Log in to comment