1. Azimuth Backup
  2. Azimuth Backup Project
  3. azimuth-inventory
  4. Issues
Issue #8 closed

NOAA Global Monitoring Division, FTP data

Jan Galkowski
created an issue

FTP data from

aftp.cmdl.noaa.gov/

to

/home/jan/local_data/aftp.cmdl.noaa.gov.global.monitoring.division

on server #2

Comments (35)

  1. Jan Galkowski reporter
    • changed status to open

    This has been in progress for about a week, and is substantially done, at least to my local box. Some of that time was the Internet outage. Will next copy to the /media/datarefuge/ directory specified above.

  2. Jan Galkowski reporter

    There are a few .odg files and PDFs which refuse to download, according to error reports and logs, because of permissions problems on the remotes. I am accepting these as lost, despite trying to get them several ways. The download log file, long as it is, records the history of these attempts and could be used to reconstruct.

  3. Jan Galkowski reporter

    It needed better organization, and I kinda forgot about it. However, it's on my list of open tickets, so am picking it up again. My EPA stuff is done. I think I suspended work when the EPA crisis erupted. I am moving the rawdata.oceanobservatories* over to azi03 (which has room), because, per ticket Issue #39, not clear if we should continue with that. Freeing up space on pub04 for the rest of this.

  4. Sakari Maaranen

    I have updated the storage use plan on pub04 according to our documented process. You should too. Plan the use as instructed, and document the space needed in the /var/local/pub/README.txt file.

  5. Jan Galkowski reporter

    This has been stopped. The lftp finally ran out of room on disk. It has been running for months, and would need to be stopped soon anyway.

    Last reports:

    lftp aftp.cmdl.noaa.gov:/aerosol> cd ../products
    cd ok, cwd=/products
    lftp aftp.cmdl.noaa.gov:/products> mirror -c . /var/local/jan/aftp.cmdl.noaa.gov.global.monitoring.division/products/
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    mirror: Login failed: 500 OOPS: cannot change directory:/ftp/aftp
    Interrupt
    lftp aftp.cmdl.noaa.gov:/products>
    

    Final size:

    [jan@pub04 jan]$ nice ionice du -s -c -b -BM /var/local/jan/aftp.cmdl.noaa.gov.global.monitoring.division/
    2679621M        /var/local/jan/aftp.cmdl.noaa.gov.global.monitoring.division/
    2679621M        total
    [jan@pub04 jan]$
    
  6. Jan Galkowski reporter

    This has been rsyncd to pub05 /var/local/jan. I have created hashes for it, but I cannot move anything until either I get permissions to copy to /varl/local/pub/DAAC-ORNL/ on pub05 or until someone else moves it for me.

  7. Log in to comment