Issue #3 resolved

Carbon Dioxide Analysis Center, Web site, http://cdiac.ornl.gov

Jan Galkowski
created an issue

Carbon Dioxide Information Analysis Center, Web site, DOE/ORNL

http://cdiac.ornl.gov

Downloaded with WGet, using:

wget --wait=20 -4 --output-file=cdiac-ornl-gov-errors.log -t 40 -nc -r -p http://cdiac.ornl.gov

Downloaded to home workstation. Putting directly on pub04 to avoid storage issues on other boxes. Will calculate SHA sums there.

Comments (23)

  1. Sakari Maaranen

    Please move these from datarefuge to ~jan/local_data/.

    0 Dec 18 02:12 cdiac.ornl.gov-pub_o
    

    Delete from datarefuge after move. Feel free to coordinate, if Scott has a better copy, as long as you get it off from datarefuge.

  2. Jan Galkowski reporter

    I don't know if Scott has any leftovers on this, or if he wants to upload his copy. If Scott has such, please upload as well, and put it in a subdirectory of the top one here.

    Thanks!

  3. marsroverdriver

    Funny enough, I also have a complete(?) snapshot of this site (downloaded on my home box, before we were properly set up with the download servers). I haven't gotten around to uploading it to the azimuth boxes yet because long story. :-/

  4. marsroverdriver

    Since this reminded me about it, I'm going ahead and transferring the ORNL site (and also a NOAA site I also downloaded to my home box) up to the download servers. It's close to 200GB of data in toto, so at the current upload rate of ~ 1MB/sec, this'll take a couple of days ....

  5. Sakari Maaranen

    marsroverdriver can you upload parts of it to different servers, or is the bandwidth constraint at your end? Our servers should be much faster than that. On the other hand, we are not in a hurry, since you already have the data, so there's no risk of us losing it. Uploading it slow but steady may be actually the best approach.

  6. John Baez

    This issue is listed as closed, but the last report here suggests it's still in process - Jan got a copy, was worried that it wasn't complete, then Scott started uploading a copy that might be complete... but is it?

    I'll re-open this. Please re-close it when it's really done, and say how many gigabytes it is, so I can add all that information to Our progress. Right now the best estimate seems to be 200 gigabytes.

  7. Sakari Maaranen

    Are all the directories under pub04:/var/local/pub/ now final? Please make sure you are not downloading directly to /var/local/pub/.

    All ongoing work on pub* servers goes in /var/local/{username}/. Only make atomic moves of completed data sets to /var/local/pub/ after completed, and report them.

    We do not want work in progress appear under pub/.

  8. Log in to comment