Issue #85 open
Jan Galkowski
created an issue

https://coast.noaa.gov/digitalcoast/

"The Digital Coast was developed to meet the unique needs of the coastal management community. The website provides not only coastal data, but also the tools, training, and information needed to make these data truly useful. Content comes from many sources, all of which are vetted by NOAA.

"Data sets range from economic data to satellite imagery. The site contains visualization tools, predictive tools, and tools that make data easier to find and use. Training courses are available online or can be brought to the user’s location. Information is also organized by focus area or topic."

Comments (22)

  1. Jan Galkowski reporter

    Getting size estimate of /pub by doing:

     lftp ftp.coast.noaa.gov/pub/
    

    and then:

    lftp ftp.coast.noaa.gov:/pub> du -s -c -b .
    

    Already in progress so please don't start a second. Want to be nice to ftp.coast.noaa.gov.

    Once we know how big this is, can download by doing:

    lftp ftp.coast.noaa.gov
    lftp ftp.coast.noaa.gov:/pub> mirror --continue . /var/local/.../ftp.coast.noaa.gov-ftp
    

    where the last part of that depends upon the server you are on and the directory structure.

    We certainly want this FTP directory, no matter.

  2. Jan Galkowski reporter

    My standard incant for doing a Web site looks like:

    httrack "http://www.charts.noaa.gov/RNCs/RNCs.shtml" -O . --mirror --depth=8 --ext-depth=3 --max-rate=100000000 %c500 --sockets=30 --retries=30 --host-control=0 TN 60 --near --robots=0 %s
    
  3. Benjamin Rose

    @Jan Galkowski here's all of the resources listed on their main site. Some are garbage, but you might like to take a quick look and see if any others might be worth keeping as part of this:

    [mirror@mirror ~]$ ./parse_json_for_sources.py
    I've identified the following assets belonging to mentioned datasets:
    Acoustic Backscatter Imagery: https://coast.noaa.gov/htdata/raster1/
    Acoustic Bathymetry, non-Charting: https://coast.noaa.gov/htdata/lidar1_z/
    American Community Survey Five-Year Estimates: https://geozoneblog.wordpress.com/2015/07/27/a-tale-of-two-coasts/
    American Community Survey Five-Year Estimates: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-county-definitions.pdf
    American Community Survey Five-Year Estimates: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coast-complicated.pdf
    American Community Survey Five-Year Estimates: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-american-community-description.pdf
    American Community Survey Five-Year Estimates: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-economy-dictionary.pdf
    Benthic Cover: https://coast.noaa.gov/data/digitalcoast/pdf/redfish-bay-mapping-report.pdf
    Benthic Cover: https://coast.noaa.gov/digitalcoast/training/cmecs-pub.html
    Benthic Cover: https://coast.noaa.gov/data/digitalcoast/pdf/bhm-guide.pdf
    Benthic Cover: https://coast.noaa.gov/data/digitalcoast/pdf/oyster-mapping.pdf
    Benthic Cover: https://coast.noaa.gov/data/digitalcoast/pdf/aerial-photography.pdf
    Benthic Cover: https://coast.noaa.gov/data/digitalcoast/pdf/satellite-multi-imaging.pdf
    Benthic Single-Beam Acoustic Soundings: https://coast.noaa.gov/digitalcoast/publications/cmecs
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-product-page.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/xls/ccap-changedata-highres.xlsx
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-faq-highres.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: http://www.asprs.org/GISD-Division/Online-Seminars.html
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/zip/ccap-mapping-bndry-wgs84.zip
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-class-scheme-regional.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/xls/ccap-changedata-highres.xlsx
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-faq-regional.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-class-scheme-highres.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-arcgis-legend.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-clipping.pdf
    Coastal Change Analysis Program High Resolution Land Cover and Change: http://www.asprs.org/GISD-Division/Online-Seminars.html#past-seminars
    Coastal Change Analysis Program High Resolution Land Cover and Change: http://oceanservice.noaa.gov/podcast/supp_july09.html#landcover
    Coastal Change Analysis Program Regional Forest Fragmentation: http://www.forestthreats.org/products/publications/fhm/kurt-riitters/A_national_assessment_of_green_infrastructure_and_change.pdf
    Coastal Change Analysis Program Regional Forest Fragmentation: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-product-page.pdf
    Coastal Change Analysis Program Regional Forest Fragmentation: https://coast.noaa.gov/data/digitalcoast/pdf/forest-frag-faq.pdf
    Coastal Change Analysis Program Regional Forest Fragmentation: http://www.forestthreats.org/research/tools/landcover-maps
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/digitalcoast/training/lake-michigan-basin-land-cover-change.html
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-regional-guidance.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/digitalcoast/training/regional-reports.html
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/digitalcoast/training/regional-land-cover-change.html
    Coastal Change Analysis Program Regional Land Cover and Change: http://noaa.maps.arcgis.com/apps/MapTour/index.html?appid=3c3cd6ef622946898f46e0760239d8cd&webmap=1cb111a1550647259d420938221aa1e9
    Coastal Change Analysis Program Regional Land Cover and Change: http://noaa.maps.arcgis.com/home/item.html?id=614b4d96244047438599e8f8eca81b27
    Coastal Change Analysis Program Regional Land Cover and Change: http://noaa.maps.arcgis.com/home/item.html?id=e7eb6e9decb14c17a2fef4d36fee1714
    Coastal Change Analysis Program Regional Land Cover and Change: http://www.noaanews.noaa.gov/stories2008/20080317_landcover.html
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/zip/ccap-legend.zip
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/zip/ccap-mapping-bndry-wgs84.zip
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-product-page.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/xls/ccap-changedata.xlsx
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-arcgis-legend.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: http://www.asprs.org/GISD-Division/Online-Seminars.html
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/zip/ccap-mapping-bndry-wgs84.zip
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-class-scheme-regional.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/xls/ccap-changedata-highres.xlsx
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-faq-regional.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-arcgis-legend.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: https://coast.noaa.gov/data/digitalcoast/pdf/ccap-clipping.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: http://oceanservice.noaa.gov/education/classroom/lessons/11_cmonitor_cchange.pdf
    Coastal Change Analysis Program Regional Land Cover and Change: http://oceanservice.noaa.gov/podcast/supp_july09.html#landcover
    Coastal Change Analysis Program Regional Land Cover and Change: http://aswm.org/wetland-science/wetlands-one-stop-mapping/7068-2014-wetland-mapping-consortium-webinars
    Coastal Flood Frequency: https://csc.noaa.gov/arcgis/rest/services/dc_slr/Flood_Frequency/MapServer
    Coastal Flood Frequency: https://coast.noaa.gov/slrdata/
    Coastal Flood Frequency: https://coast.noaa.gov/digitalcoast/tools/slr
    Coastal IfSAR and other derived DEMs from NOAA OCM: https://coast.noaa.gov/htdata/lidar1_z/
    Coastal IfSAR and other derived DEMs from NOAA OCM: https://coast.noaa.gov/htdata/raster1/
    Coastal Topobathy Lidar (JALBTCX): https://coast.noaa.gov/geozone/bathy-lidar-harder-than-it-looks/
    Coastal Topobathy Lidar (JALBTCX): http://www.ngs.noaa.gov/corbin/class_description/Lidar_presentations.shtml
    Coastal Topobathy Lidar (JALBTCX): http://coastal.er.usgs.gov/lsrm/tech/tech1-eaarl.html
    Coastal Topobathy Lidar (JALBTCX): http://shoals.sam.usace.army.mil/
    Coastal Topobathy Lidar (JALBTCX): https://coast.noaa.gov/htdata/lidar1_z/
    Coastal Topographic Lidar: https://coast.noaa.gov/geozone/taking-stock-of-all-that-lidar/
    Coastal Topographic Lidar: https://coast.noaa.gov/geozone/lidar-point-clouds/
    Coastal Topographic Lidar: https://coast.noaa.gov/geozone/first-return-lidar-in-digital-coast/
    Coastal Topographic Lidar: https://coast.noaa.gov/geozone/too-many-government-elevation-sites/
    Coastal Topographic Lidar: http://rapidlasso.com/blog/
    Coastal Topographic Lidar: http://lidarnews.com/
    Coastal Topographic Lidar: https://csc.noaa.gov/mailman/listinfo/coastal-elevation
    Coastal Topographic Lidar: http://www.asi-gis.com/2013/08/ortho-lidar-accuracy-concepts-webinar-july-31-2013-video-slides-available/
    Coastal Topographic Lidar: http://wrc.umn.edu/randpe/agandwq/tsp/lidar/
    Coastal Topographic Lidar: https://coast.noaa.gov/data/digitalcoast/pdf/lidar-provisioning-guidance.pdf
    Coastal Topographic Lidar: http://www.geonext.nl/wp-content/uploads/2014/05/Point-Cloud-Viewers.pdf
    Coastal Topographic Lidar: https://coast.noaa.gov/data/digitalcoast/pdf/lidar-101.pdf
    Coastal Topographic Lidar: https://www.esri.com/~/media/Files/Pdfs/library/workshops/arcgis102-lidar-workshop.pdf
    Coastal Topographic Lidar: https://coast.noaa.gov/data/digitalcoast/pdf/lidar-arcgis-tutorial.pdf
    Coastal Topographic Lidar: https://coast.noaa.gov/data/digitalcoast/pdf/lidar-marshes-slamm.pdf
    Coastal Topographic Lidar: https://coast.noaa.gov/htdata/lidar1_z/
    Coastal Topographic Lidar: http://www.oregongeology.org/sub/projects/olc/gsa2009_course515.htm
    Coastal Topographic Lidar: https://coast.noaa.gov/digitalcoast/training/intro-lidar
    Critical Facilities: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-county-definitions.pdf
    Critical Facilities: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-critical-facilities-description.pdf
    Critical Facilities: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-critical-facilities-dictionary.pdf
    Decadal Demographic Trends in the U.S.: https://geozoneblog.wordpress.com/2015/07/27/a-tale-of-two-coasts/
    Decadal Demographic Trends in the U.S.: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-county-definitions.pdf
    Decadal Demographic Trends in the U.S.: http://oceanservice.noaa.gov/facts/coastal-population-report.pdf
    Decadal Demographic Trends in the U.S.: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coast-complicated.pdf
    Decadal Demographic Trends in the U.S.: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-demographic-trends-description.pdf
    Decadal Demographic Trends in the U.S.: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-demographic-trends-dictionary.pdf
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/enow-product-page.pdf
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/category/econ/
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/enow-plus/#.VMfmbv54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/economy-fascinating-facts/#.VMfdZ_54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/fun-sun/#.VMfmHv54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/gdp-real-gdp-and-chained-2005-dollars/#.VMffoP54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/ocean-and-great-lakes-economy/#.VMfnV_54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/speed-dating-with-the-coastal-and-ocean-economies/#.VMflvf54rMo
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/suppression-mystery-vanishing-data/#.VMffQv54rMo
    Economics: National Ocean Watch: https://geozoneblog.wordpress.com/2015/06/08/ocean-economy-big-and-growing/
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/bigger-ocean-economy/
    Economics: National Ocean Watch: https://coast.noaa.gov/geozone/whats-in-a-job/#.VMffB_54rMo
    Economics: National Ocean Watch: https://geozoneblog.wordpress.com/2015/07/27/a-tale-of-two-coasts/
    Economics: National Ocean Watch: http://noaa.maps.arcgis.com/apps/MapTour/index.html?appid=f9efd19854294f9295a2f064ef6c2e2b&webmap=fbc594d6ca0a424bbd2af9c641fcea4f
    Economics: National Ocean Watch: http://oceanservice.noaa.gov/podcast/june15/dd61-oceaneconomy.html
    Economics: National Ocean Watch: https://coast.noaa.gov/digitalcoast/training/?filter=Coastal+Economy;Videos+and+Webinars
    Economics: National Ocean Watch: http://noaa.maps.arcgis.com/apps/MapTour/index.html?appid=81bebea1299f4521ba68ecffd6dc17b5&webmap=19ed1fd7888043fbb27a23400e5fe48b
    Economics: National Ocean Watch: http://noaacsc.adobeconnect.com/p435jgft6ml/
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/econ-report.pdf
    Economics: National Ocean Watch: http://noaa.maps.arcgis.com/apps/StorytellingSwipe/index.html?appid=c8f844ebb93f4d598c8a24cb7934b083&webmap=849dab50b6e74a64aa002c5a93f2cdaa
    Economics: National Ocean Watch: http://noaa.maps.arcgis.com/apps/MapJournal/?appid=639339405df9482ea2a907038fa75eb0
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/enow-faq.pdf
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/enow-counties-list.pdf
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/enow-crosswalk-table.pdf
    Economics: National Ocean Watch: https://coast.noaa.gov/data/digitalcoast/pdf/enow-infographic.pdf
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/geozone/enow-plus/#.VMfmbv54rMo
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/geozone/economy-fascinating-facts/#.VMfdZ_54rMo
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/geozone/ocean-and-great-lakes-economy/#.VMfnV_54rMo
    Economics: National Ocean Watch for Self-Employed Workers: http://noaa.maps.arcgis.com/apps/MapTour/index.html?appid=f9efd19854294f9295a2f064ef6c2e2b&webmap=fbc594d6ca0a424bbd2af9c641fcea4f
    Economics: National Ocean Watch for Self-Employed Workers: http://oceanservice.noaa.gov/podcast/june15/dd61-oceaneconomy.html
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/digitalcoast/training/?filter=Coastal+Economy;Videos+and+Webinars
    Economics: National Ocean Watch for Self-Employed Workers: http://noaa.maps.arcgis.com/apps/MapTour/index.html?appid=81bebea1299f4521ba68ecffd6dc17b5&webmap=19ed1fd7888043fbb27a23400e5fe48b
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/data/digitalcoast/pdf/econ-report.pdf
    Economics: National Ocean Watch for Self-Employed Workers: http://noaa.maps.arcgis.com/apps/MapJournal/?appid=639339405df9482ea2a907038fa75eb0
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/data/digitalcoast/pdf/enow-nes-crosswalk-table.pdf
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/data/digitalcoast/pdf/enow-counties-list.pdf
    Economics: National Ocean Watch for Self-Employed Workers: https://coast.noaa.gov/data/digitalcoast/pdf/enow-infographic.pdf
    Economics: National Ocean Watch for Self-Employed Workers: http://www.census.gov/econ/nonemployer/methodology.htm
    Gross Domestic Product by State: http://www.bea.gov/regional/docs/product/readmeNAICS.cfm
    Gross Domestic Product by State: http://www.bea.gov/regional/docs/product/readmeSIC.cfm
    Gross Domestic Product by State: http://www.bea.gov/methodologies/index.htm
    High-Resolution Orthoimagery: https://coast.noaa.gov/htdata/raster1/
    Historical Aerial Imagery: https://coast.noaa.gov/htdata/raster1/
    Integrated Ocean Observing System (IOOS): http://www.osmc.noaa.gov/
    Integrated Ocean Observing System (IOOS): http://www.ioos.noaa.gov/ioos_in_action/stories/welcome.html
    Integrated Ocean Observing System (IOOS): http://oceanservice.noaa.gov/podcast/supp_mar09.html#ioos
    Integrated Ocean Observing System (IOOS): http://howsthebeach.org/
    Integrated Ocean Observing System (IOOS): http://www.ioos.noaa.gov/modeling/galleries.html
    Integrated Ocean Observing System (IOOS): http://www.ioos.noaa.gov/qartod
    Integrated Ocean Observing System (IOOS): http://www.socat.info/
    Integrated Ocean Observing System (IOOS): http://ioos.github.io/
    Landsat Satellite Imagery: http://landsatlook.usgs.gov/viewer.html
    Landsat Satellite Imagery: http://landsat.usgs.gov/gallery.php
    Landsat Satellite Imagery: http://landsat.usgs.gov/tools_faqs.php
    Landsat Satellite Imagery: http://landsat.gsfc.nasa.gov/?page_id=2372
    Landsat Satellite Imagery: http://landsat.usgs.gov/links_educationallinks.php
    Landsat Satellite Imagery: http://landsat.gsfc.nasa.gov/
    Landsat Satellite Imagery: http://landsat.usgs.gov/
    Lidar-Derived Bottom Reflectance Imagery: https://coast.noaa.gov/htdata/lidar1_z/
    Lidar-Derived Bottom Reflectance Imagery: https://coast.noaa.gov/htdata/raster1/
    Marine Resources Geographic Information System Internet Map Server: http://www.arcgis.com/home/item.html?id=a75c5abd603044efb24c6e3e99623eaf
    Marine Resources Geographic Information System Internet Map Server: http://ocean.floridamarine.org/mrgis/Description_Layers_Marine.htm
    Marine Resources Geographic Information System Internet Map Server: http://ocean.floridamarine.org/mrgis/
    National Elevation Dataset: http://ned.usgs.gov
    National Estuarine Research Reserve System-Wide Monitoring Program Data: http://cdmo.baruch.sc.edu/contact.cfm
    National Estuarine Research Reserve System-Wide Monitoring Program Data: http://www.nerrsdata.org
    National Estuarine Research Reserve System-Wide Monitoring Program Data: http://cdmo.baruch.sc.edu/get/realTime.cfm
    National Hydrography Dataset: http://viewer.nationalmap.gov/viewer/nhd.html?p=nhd
    National Hydrography Dataset: http://nhd.usgs.gov/applications.html#.VMt46GjF-Ah
    National Hydrography Dataset: http://nhd.usgs.gov/Frequently+Asked+Questions+about+the+NHD+&+WBD.htm
    National Hydrography Dataset: http://nhd.usgs.gov/index.html
    National Hydrography Dataset: http://nhd.usgs.gov/wbd.html
    National Hydrography Dataset: http://pubs.usgs.gov/fs/2009/3054/
    National Land Cover Database: http://www.mrlc.gov
    National Land Cover Database: http://pubs.usgs.gov/fs/2012/3020/fs2012-3020.pdf
    Nonemployer Statistics: https://ask.census.gov/
    Nonemployer Statistics: http://www.census.gov/econ/nonemployer/methodology.htm
    Nonemployer Statistics: http://www.census.gov/eos/www/naics/
    OpenTopography: http://www.opentopography.org/index.php/blog
    OpenTopography: http://www.opentopography.org
    Quarterly Census of Employment and Wages: http://www.bls.gov/cew/publications.htm
    Quarterly Census of Employment and Wages: http://www.bls.gov/news.release/cewqtr.toc.htm
    Quarterly Census of Employment and Wages: http://www.bls.gov/cew/cewfaq.htm
    Quarterly Census of Employment and Wages: http://www.census.gov/eos/www/naics/
    Sea Level Rise Inundation: https://coast.noaa.gov/digitalcoast/tools/slr
    Social Vulnerability Index 2010 Census Tracts: http://webra.cas.sc.edu/hvri/products/sovi_32.aspx
    Total Economy of Coastal Areas: https://geozoneblog.wordpress.com/2015/07/27/a-tale-of-two-coasts/
    Total Economy of Coastal Areas: https://coast.noaa.gov/geozone/speed-dating-with-the-coastal-and-ocean-economies/#.VMaWYyvF_To
    Total Economy of Coastal Areas: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-county-definitions.pdf
    Total Economy of Coastal Areas: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coast-complicated.pdf
    Total Economy of Coastal Areas: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-economy-description.pdf
    Total Economy of Coastal Areas: https://coast.noaa.gov/data/digitalcoast/pdf/qrt-coastal-economy-dictionary.pdf
    USDA Soil Survey Geographic Database: http://www.nrcs.usda.gov/wps/portal/nrcs/detail/soils/survey/?cid=nrcs142p2_053627
    USDA Soil Survey Geographic Database: http://websoilsurvey.nrcs.usda.gov/DataAvailability/SoilDataAvailabilityMap.pdf
    USFWS National Wetlands Inventory: http://www.fws.gov/wetlands/Status-and-Trends/index.html
    USFWS National Wetlands Inventory: http://www.fws.gov/wetlands/FAQs.html
    USFWS National Wetlands Inventory: http://www.fws.gov/wetlands/index.html
    
  4. Jan Galkowski reporter

    I have checked in a markup of the above in BitBucket's Git azimuth-inventory, under the folder for coast.noaa.gov_digitalcoast showing which (most) are worth keeping. Thanks.

  5. Jan Galkowski reporter

    Believe it or not, that lftp and du are still running without completion this morning. So my guess is that the datasets are truly enormous. I continue to think they are very important. I'm just grateful we have Benjamin's store to rely upon. Don't know where we'd put them on our servers.

  6. Benjamin Rose

    Seems like wget finished, but the size is perhaps smaller than I was expecting:

    FINISHED --2017-01-25 20:20:15-- Total wall clock time: 1d 22h 23m 27s Downloaded: 79109 files, 690G in 1d 9h 58m 10s (5.78 MB/s)

  7. Benjamin Rose

    Same deal here, my initial wget didn't get everything, I had an error in my script. It's picking up the rest now, there should be quite a bit more. @Jan Galkowski did that "du" ever finish yet? I can spare about 50TB so everything should fit.

  8. Jan Galkowski reporter

    BELIEVE it or not, it is STILL running! I was thinking of killing it, but I decided against it. I'll let you know as soon as I know anything. But it's an indication of how much larger your bandwidth is than ours.

  9. Benjamin Rose

    @Jan Galkowski Not entirely... you're grabbing .listing files and tallying it up, this could represent 90% of the time for me to do the same plus fetch the files over Internet2, depending on remote resources. My wget error was silly, and while for ftp.epa.gov isn't a big deal (its almost done resolving deltas), it may have missed a ton of data for this coastal set. Time shall tell!

  10. Log in to comment