1. Azimuth Backup
  2. Azimuth Backup Project
  3. azimuth-inventory
  4. Issues
Issue #18 resolved

NOAA tides and currents products Web site, including sea level trends

Jan Galkowski
created an issue

https://tidesandcurrents.noaa.gov/products.html which includes https://tidesandcurrents.noaa.gov/sltrends/sltrends.html

Replicate

https://tidesandcurrents.noaa.gov/products.html

to Sakari's Datarefuge FTP server at

tidesandcurrents.noaa.gov.products

using

wget --dns-timeout=10 --connect-timeout=20 --read-timeout=120 --wait=12 --random-wait --prefer-family=IPv4  --tries=40 --timestamping=on --recursive --level=8 --no-remove-listing  --follow-ftp -nv --output-file=tidesandcurrents-noaa-gov-products.log --no-check-certificate  https://tidesandcurrents.noaa.gov/products.html

Comments (19)

  1. Jan Galkowski reporter

    Had to suspend this one for a bit because needed to move the target to a different disk, since the previous hosting disk was running out of room. Will restart once at the new disk.

  2. Jan Galkowski reporter

    This is in progress. It has been running since 4 January 2017 on azi03, and has collected a little over 6 Gb of data there, in /home/jan/local_data/tidesandcurrents.noaa.gov. It pulls a new file about every 8 seconds. As near as I can tell, the source server is just very slow, and I keep my --wait=5 with a *--random-wait * to avoid getting throttled more.

    • Jan
  3. Jan Galkowski reporter

    Greg Kochanski I could not set_read_only.sh on Pub01 /var/local/pub/TIDES-AND-CURRENTS. BTW, I found out why I was getting line endings wrong on the transfer. I first suspected WinSCP, but I was transferring binary, so that could not be it. It turns out that when I use SourceTree and pull set_read_only.sh down, the line endings at my machine are already changed to CRLF. I always keep my endings LF in my editor and files on my Windows machine, because I use 'nixen so much. So I can fix it, but I was trying to fix it in the wrong place.

    Anyway, this still isn't read-only.

  4. Sakari Maaranen

    This has now been set read-only.

    [root@pub01 pub]# du -sbc NOAA_Tides_and_Currents/2017-02-03_18/*
    3342965610      NOAA_Tides_and_Currents/2017-02-03_18/tidepool.nos.noaa.gov
    3812993230      NOAA_Tides_and_Currents/2017-02-03_18/tidesandcurrents.noaa.gov
    5086877 NOAA_Tides_and_Currents/2017-02-03_18/tidesandcurrents-noaa-gov-products.log.gz
    1512034 NOAA_Tides_and_Currents/2017-02-03_18/tides_packup.log.gz
    7162557751      total
    
  5. Log in to comment