1. Azimuth Backup
  2. Azimuth Backup Project
  3. azimuth-inventory

Wiki

Clone wiki

azimuth-inventory / Our progress

#8

Our progress

For up-to-date news see the RESOLVED and COMPLETE issues on the issue tracker.

In the table below:

  • Always report the size in gigabytes, so all the numbers remain comparable.
  • Always check the origin site terms of use, licencing policy, and/or intellectual property rights status. Provide link to their public statement or obtain permission from the origin site administrators and attach the permission letter to the relevant item in the issue tracker.

Do not double report information that is already in the issue tracker. Otherwise maintaining the report itself becomes a lot of work. The purpose of this page is to give an overview at a single glance.

Backups on the Azimuth storage space

Size/GiB Data set Description
1721 GB University of Idaho Gridded Surface Meteorological Data (UofI METDATA) #1
1261 GB NASA PDS: The Planetary Data System #23
103 GB (issue #9), 154 GB & 35880 files (issue #65) NASA GISTEMP website at http://data.giss.nasa.gov/gistemp/ NASA is a bit spread out, but I think their climate histories are here: http://data.giss.nasa.gov/ and their assimilation/analysis products are here: https://cds.nccs.nasa.gov/data/ #9. "This was one of my earliest attempts at replication using wget and so is probably imperfect." Also overlapping download #65 that covers https://cds.nccs.nasa.gov/data/ (some small amount of missing data).
186 GB NOAA Carbon Dioxide Information Analysis Center (CDIAC) data at ftp://ftp.ncdc.noaa.gov/pub/ #4.
298 GB NOAA Carbon Dioxide Information Analysis Center (CDIAC) data at https://cdiac.ornl.gov #3 "The Carbon Dioxide Information Analysis Center (CDIAC), located at the U.S. Department of Energy's (DOE) Oak Ridge National Laboratory (ORNL), is the primary climate change data and information analysis center for DOE. CDIAC is supported by DOE's Climate and Environmental Sciences Division within the Office of Biological and Environmental Research (BER)."
15 GB NOAA Carbon Tracker website at http://www.esrl.noaa.gov/psd/data/gridded/data.carbontracker.html Issues #2 "Done with httrack.""
298 GB NOAA Carbon Tracker website at http://www.esrl.noaa.gov/psd/data/gridded/data.carbontracker.html Issues #12 "Done with wget."
109 GB OAR datasets from NOAA Research at ftp://ftp.oar.noaa.gov/ #22. SHA-256 generated.
29 GB U.S. Forest Service Climate Change Atlas website at http://www.fs.fed.us/nrs/atlas/ #15 "Completed, with much time spent comparing SHA256 reports."
513 GB NOAA ESRL CT2015 Carbon Tracker website at https://www.esrl.noaa.gov/gmd/ccgg/carbontracker/ #13 "Ach! Tried to do a SHA256 recursively on the files in the Windows side of the world. No luck." "Completed, with discrepancies logged, and no SHA256 calculated."
22 GB U.S. Energy Information Administration #48 "ftp.eia.gov + www.eia.gov" : "The U.S. Energy Information Administration (EIA) collects, analyzes, and disseminates independent and impartial energy information to promote sound policymaking, efficient markets, and public understanding of energy and its interaction with the economy and the environment."
720 GB NOAA NCEP/NCAR Reanalysis at ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/ [#6] "The NCEP/NCAR Reanalysis 1 project is using a state-of-the-art analysis/forecast system to perform data assimilation using past data from 1948 to the present." Some issues remain with different downloads giving different sets of files.
220 GB Carbon Tracker, tracking CO2 and CH4 emissions #7 "ftp://aftp.cmdl.noaa.gov/products/carbontracker/ : "
301 GB ftp.daac.ornl.gov, www.daac.ornl.gov #26, #27, #27 "The Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC) for biogeochemical dynamics is one of the NASA Earth Observing System Data and Information System (EOSDIS) data centers managed by the Earth Science Data and Information System (ESDIS) Project"
25 GB U.S. NOAA ESRL GMD data archive at http://www.esrl.noaa.gov/gmd/dv/ftpdata.html #14 "NOAA Global Monitoring Division data archive"
10.6 GB, 136180 files http://data.gov/climate a central website full of pointers (little/no data) #66. "You can currently find data and resources related to coastal flooding, food resilience, water, ecosystem vulnerability, human health, energy infrastructure, transportation and the Arctic region." Uploaded by Greg.
2.5 GB, 6040 files. USDA Climate-related web pages and tools. drought.gov, climatehubs.usda.gov, *.fs.usda.gov and other bits. #82
0.02 GB, 470 files USDA climate-related blog and links. www.usda.gov, blogs.usda.gov #76
94 GB NOAA's National Centers for Environmental Information (NCEI), Web site #5; "NOAA's National Centers for Environmental Information (NCEI) is responsible for preserving, monitoring, assessing, and providing public access to the Nation's treasure of climate and historical weather data and information."
13404 GB NOAA's National Centers for Environmental Information (NCEI), FTP site #10 "NOAA's National Centers for Environmental Information (NCEI) is responsible for preserving, monitoring, assessing, and providing public access to the Nation's treasure of climate and historical weather data and information."
84 GB U.S. DOE Office of Energy Efficiency and Renewable Energy #46 "https://energy.gov/eere/office-energy-efficiency-renewable-energy; As part of its work to support energy efficiency and renewable energy research, the Office of Energy Efficiency and Renewable Energy has collaborated with private organizations, researchers, and other nations. Here you can read about the results of EERE's collaboration and research activities. This list includes projects completed at the office level, or by the Business Operations or Strategic Programs areas."
2 GB U.S. DOE Transportation Energy Data Book #47 "http://cta.ornl.gov/data/index.shtml ; Edition 35 Released October 31, 2016. The Transportation Energy Data Book (TEDB) is a compendium of data on transportation with an emphasis on energy. Designed for use as a desktop reference, the TEDB was first published in 1976 and has continued to Edition 35. The TEDB is produced by Oak Ridge National Laboratory for the U.S. Department of Energy's Office of Energy Efficiency and Renewable Energy"
24 GB Sealevel mission, JASON3 NASA JPL #16 "http://sealevel.jpl.nasa.gov/missions/jason3/ ; Jason-3 is the fourth mission in U.S.-European series of satellite missions that measure the height of the ocean surface. Launched on January 17, 2016, the mission will extend the time series of ocean surface topography measurements (the hills and valleys of the ocean surface) begun by the TOPEX/Poseidon satellite mission in 1992 and continuing through the Jason-1 (launched in 2001) and the currently operating OSTM/Jason-2 (launched in 2008) missions."
1529 GB climate.nasa.gov #45
888M, 722 files Region of climate.nasa.gov near http://climate.nasa.gov/scientific-consensus Issue #97
509 GB data.noaa.gov #36
5 GB lpdaac #71
132 GB noaa-sea-surface-temps #35
? MODIS, https://modis.gsfc.nasa.gov/data/dataprod/ #34: Records indicate downloaded, but trying to find datasets so can complete documentation (jtg)
25 GB water.usgs.gov #72
? globalchange.gov #44
872 GB (STILL IN PROGRESS) podaac #33
218 GB www.wcc.nrcs.usda.gov #73
5 GB NASA Ocean Color Data http://oceandata.sci.gsfc.nasa.gov #24 "Ocean Biology (OB) data produced or collected under NASA’s Earth Observing System Data and Information System (EOSDIS)."
26 GB https://energy.gov/epsa/office-energy-policy-and-systems-analysis #84 "The Director of the Office of Energy Policy and Systems Analysis is the primary energy policy advisor to the Secretary and Deputy Secretary on domestic energy policy development and implementation as well as DOE policy analysis and activities. The role of the Office of Energy Policy and Systems Analysis is to deliver unbiased energy analysis to the Department of Energy's leadership on existing and prospective energy-related policies, focusing in part on integrative analysis of energy systems."
3015 GB NOAA global ARGO floats data (https://www.nodc.noaa.gov/argo/, https://www.nodc.noaa.gov/argo/accessData.htm, ftp://ftp.nodc.noaa.gov/pub/data.nodc/argo/, ftp://ftp.nodc.noaa.gov/nodc/archive/) #30 "The broad-scale global array of temperature/salinity profiling floats, known as Argo, has already grown to be a major component of the ocean observing system. Argo is a standard to which other developing ocean observing systems can look to. For example, Argo offers ideas on various topics such as how to collaborate internationally, how to develop a data management system and how to change the way scientists think about collecting data. Deployments began in 2000 and continue today at the rate of about 800 per year. See http://www.argo.ucsd.edu/"
110 GB NOAA National Data Buoy Center (http://www.ndbc.noaa.gov/) #11 "NDBC designs, develops, operates, and maintains a network of data collecting buoys and coastal stations. NDBC's virtual tour will give you an overview of what NDBC does, the facilities, the atmospheric and oceanographic variables measured, and the use of these data."
3736 GB NOAA NCEI Satellite Datasets Web site #17 "Satellite data listed by parameter being observed. Access to long-term, high-quality products is supported by NOAA's Climate Data Record (CDR) Program."
16 GB (295000 files) http://climate.gov and closely related sites. #60 This is a central set of pointers to lots and lots of websites. This download has no data, just links and documents. (20 Jan 2017).
18 GB (81601 files, excluding checksums) https://cpo.noaa.gov Climate Project Office #111
43 GB (40060 files) NOAA Precipitation Frequency Data Server website and ftp site at http://hdsc.nws.noaa.gov/hdsc/pfds/ and ftp://hdsc.nws.noaa.gov/pub #21
292 GB Carbon Tracker Web site, ESRL NOAA, gridded at http://www.esrl.noaa.gov/psd/data/gridded/data.carbontracker.html [#12]
12 MB 274 files EPA superfund pages http://www.epa.gov #41; Collected 2016-12-26.
268 GB The EPA website, centered on climate change pages. http://www.epa.gov/climatechange and vicinity. #78 Downloaded Jan 19, 2017.
124 GB The EPA waste reduction model, WARM. http://www.epa.gov/warm. #88 Downloaded 10 February 2017.
16GB, 28744 files https://www.epa.gov/air-emissions-inventories and vicinity. #93. The National Emissions Inventory (NEI) is a detailed estimate of air emissions that include criteria pollutants and hazardous air pollutants. Note we had no access to the database.
0.82 GB, 27300 files geodata.epa.gov, misc collection. #62
20 GB, 18254 files nrel.gov National Renewable Energy Laboratory #95
306G, 381872 files https://www.cdc.gov Centers for Disease Control and Prevention. #94 Collected 2017-02-05.
6 GB http://icebridge.nasa.gov, Operation Ice Bridge Issue#98 "NASA’s Operation IceBridge images Earth’s polar ice in unprecedented detail to better understand processes that connect the polar regions with the global climate system. "
7 GB https://tidesandcurrents.noaa.gov/products.html, NOAA tides and currents products Web site, including sea level trends Issue #18 "NOAA's Tides and Currents website, developed and supported by the Center for Operational Oceanographic Products and Services (CO-OPS), provides the following products. A definition of each term and a description of each product are provided."
38 GB 106764 files http://phmsa.dot.gov Pipeline and Hazardous Materials Safety Administration Issue #100 2017-02-02T2102Z
2679 (mostly done: ran out of time and room) NOAA Global Monitoring Division, FTP data NOAA Issue #8 "FTP data at GMD"
18 GB NASA Atmospheric Science Data Center (ASDC) #29
23 Gb DSCOVR satellite data www.ngdc.noaa.gov/dscovr Issue #110 "DSCOVR satellite data through p20170317033324"
76 Gb www.epa.gov database (https://www.epa.gov/enviro/envirofacts-model, https://www.epa.gov/enviro/greenhouse-gas-restful-data-service) Issue #79 "www.epa.gov database"

Backups on Scott Maxwell's Google Drive

Size/GiB Data set Description
132 GB NOAA sea surface temperature data at ftp://data.ncdc.noaa.gov/cdr/sea-surface-temp-whoi/files/ #35. SHA-256 checksums made.

Updated