1. Azimuth Backup
  2. Azimuth Backup Project
  3. azimuth-inventory
  4. Issues
Issue #17 open

NOAA NCEI Satellite Datasets Web site

Jan Galkowski
created an issue

https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasets

Replicate

https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasets

to Sakari's Datarefuge FTP server in

ncdc.noaa.gov.data-access.satellite-data.satellite-data-access-datasets

using

wget --dns-timeout=10 --connect-timeout=20 --read-timeout=120 --wait=12 --random-wait --prefer-family=IPv4  --tries=40 --timestamping=on --recursive --level=8 --no-remove-listing  -nv --output-file=ncdc-noaa-satellite-access-datasets-gov.log --follow-ftp --no-check-certificate  https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasets

Comments (11)

  1. Jan Galkowski reporter

    This is being copied from the site you specify, still in progress. Plan as usual is to download it, and then transfer to Sakari's FTP server. This is going to my NAS. It currently has 57 GB in 9900+ files.

  2. Jan Galkowski reporter

    marsroverdriver https://bitbucket.org/John_Baez/ I do not know. In fact, we do not know. We know little of the semantics of these files, and, moreover, the master list of files was a survey and no attempt was made to identify duplication. Moreover, we've been told that even the master list at https://goo.gl/4ffTc7 is both incomplete and redundant.

    Accordingly, given the situation, My philosophy on the matter is to simply replicate, since we can't discriminate. I'd rather have a couple of copies than no copies. Moreover, that's how the overall project is trying to constrain errors, although I don't know what people would do with file naming problems except, perhaps, to use different tools.

  3. Jan Galkowski reporter

    Had to suspend this one for a bit because needed to move the target to a different disk, since the previous hosting disk was running out of room. Will restart once at the new disk.

  4. Jan Galkowski reporter

    Ach! Going way too slow to the offline disk. Not making progress at all. Will restart on the server directly, hoping to make up time. Would have had to transfer by FTP from local disk to the server anyway.

    Destination is now:

    /media/jan-one/ncdc.noaa.gov.data-access.satellite-data.satellite-data-access-datasets
    
  5. Log in to comment