University of Idaho Gridded Surface Meteorological Data (UofI METDATA)

Create issue
Issue #1 resolved
Sakari Albert Maaranen created an issue

2016-12-17 Hetzner storage box in Germany

wget --wait=12 --random-wait --prefer-family=IPv4 –verbose=on \
 --dns-timeout=10 --connect-timeout=20 --read-timeout=120 \
 --tries=40 --timestamping=on --recursive --level=inf \
 --no-remove-listing –output-file=climate.nkn.uidaho.edu_METDATA.log \
 --follow-ftp --no-check-certificate \
 -H -Dclimate.nkn.uidaho.edu,northwestknowledge.net \
 http://climate.nkn.uidaho.edu/METDATA/ \
 https://www.northwestknowledge.net/metdata/data/ \
 http://thredds.northwestknowledge.net:8080/thredds/reacch_climate_MET_catalog.html

Comments (18)

  1. Sakari Albert Maaranen reporter

    Abatzoglou, John (jabatzoglou@uidaho.edu) jabatzoglou@uidaho.edu wrote 19 December 2016 at 22:09:

    Hello Sakari,

    Thank you for your assistance and support of climate science research and associated datasets. I have asked Dr. Katherine Hegewisch (cc’ed here) to see if she could look into checksums for these files. We do update these data daily, so it may be fine to constrain the focus to data prior to 2016.

    You have my permission to mirror these data.

    I am not as worries about my datasets as they are on servers hosted by US universities, rather than the federal gov’t. Our data have also been shared with the Google cloud through Earth Engine in the event that helps prioritize other datasets for the refuge efforts.

    Thanks for your efforts!

    John

  2. Sakari Albert Maaranen reporter

    This backup script has already run several days and has still not ended. Probably something wrong with the wget usage. Perhaps we should focus only on the files they have hashed above.

  3. Sakari Albert Maaranen reporter

    Interrupted the download. The valuable data is probably there, but there has to be loads of unnecessary extra files. Someone who understands this data set should investigate and perhaps drop the excess. Keep at least the hashed files.

  4. Sakari Albert Maaranen reporter

    The following directories and files were created as a part of this job. @Joos-gcv must have gzipped the transfer log when the transfer was still in progress, so it may be partial.

    0 Dec 31 21:03 climate.nkn.uidaho.edu
    0 Dec 31 21:25 climate.northwestknowledge.net
    8740817 Jan  3 13:05 climate.nkn.uidaho.edu_METDATA.log.gz
    0 Dec 31 20:48 maca.northwestknowledge.net
    0 Dec 21 12:01 metdata.northwestknowledge.net
    0 Dec 18 19:41 thredds.northwestknowledge.net:8080
    0 Dec 20 23:37 www.northwestknowledge.net
    

    All the above directories and files should now be moved together, away from datarefuge. I have a disk usage command running and will report size as soon as it completes.

  5. Sakari Albert Maaranen reporter
    [sam@azi03 datarefuge]$ nice ionice du -s -c -b *
    4207353 climate.nkn.uidaho.edu
    8740817 climate.nkn.uidaho.edu_METDATA.log.gz
    48418317        climate.northwestknowledge.net
    12541147        maca.northwestknowledge.net
    467007  metdata.northwestknowledge.net
    1189464254871   thredds.northwestknowledge.net:8080
    657825603504    www.northwestknowledge.net
    
  6. Sakari Albert Maaranen reporter

    No worries. I wasn't expecting my initial download to take weeks... I started it before we had any azi## servers. Most likely, and again just a guess, the valuable parts had already been transferred and the process was churning through some dynamically generated pages. Anyway, need to inspect to make sure.

  7. Sakari Albert Maaranen reporter

    @Joos-gcv where have you moved the climate.nkn.uidaho.edu directory? I didn't ask you to move it. Doesn't matter as long as it is safe. We should keep it together with all directories mentioned here.

    I am copying the northwestknowledge directories to pub05:/var/local/sam/datarefuge/.

    Please make sure there is no overlapping work.

  8. Sakari Albert Maaranen reporter

    Also climate.nkn.uidaho.edu_METDATA.log.gz is now at pub05:/var/local/sam/datarefuge/

  9. Sakari Albert Maaranen reporter

    Nevermind it. Found it and moved it to pub05. Once the copying of *northwestknowledge* finishes, these data will then live all in the same place.

  10. Sakari Albert Maaranen reporter
    sent 1847595663018 bytes  received 2715273 bytes  60901471.06 bytes/sec
    total size is 1847351284846  speedup is 1.00
    

    The command was:

    [sam@pub05 ~]$ ionice -c 2 -n 5 rsync -avub /media/datarefuge/*northwestknowledge* /var/local/sam/datarefuge
    
  11. Sakari Albert Maaranen reporter
    • changed status to resolved
    • edited description

    Not sure if this is complete, but it is a lot.

    [sam@pub05 ~]$ du --apparent-size --summarize --total -BG /var/local/sam/*
    1G      /var/local/sam/climate.nkn.uidaho.edu
    1G      /var/local/sam/climate.nkn.uidaho.edu_METDATA.log.gz
    1G      /var/local/sam/climate.northwestknowledge.net
    1G      /var/local/sam/maca.northwestknowledge.net
    1G      /var/local/sam/metdata.northwestknowledge.net
    1108G   /var/local/sam/thredds.northwestknowledge.net:8080
    613G    /var/local/sam/www.northwestknowledge.net
    1721G   total
    
  12. Log in to comment