tsgettoolbox: downloading from LDAS

[Ritvik Sahajpal]

Hi Tim,

Many thanks for an excellent package. This is exactly what the community needs.

I had a question about the LDAS download part. While it works for points in USA (where presumably NLDAS is invoked), it does not work for global points e.g. the foll. code fails with a HTTP Error 404: Not found

from tsgettoolbox import tsgettoolbox
df = tsgettoolbox.ldas(lat=35.86,
                        lon=104.2,
                        variable='GLDAS:GLDAS_NOAH025_3H.001',
                        startDate='2016-01-01',
                        endDate='2017-01-01')

Is this because a user account needs to be created for GLDAS?

Error is below:

  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\tsgettoolbox\tsgettoolbox.py", line 1097, in ldas
    return tsutils.printiso(odo(r, pd.DataFrame))
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\odo\odo.py", line 91, in odo
    return into(target, source, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\odo\into.py", line 43, in wrapped
    return f(*args, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\odo\into.py", line 53, in into_type
    return convert(a, b, dshape=dshape, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\odo\core.py", line 44, in __call__
    return _transform(self.graph, *args, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\odo\core.py", line 60, in _transform
    x = f(x, excluded_edges=excluded_edges, **kwargs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\tsgettoolbox\services\ldas.py", line 111, in ldas_to_df
    delim_whitespace=True)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 375, in _read
    compression=kwds.get('compression', None))
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\common.py", line 238, in get_filepath_or_buffer
    req = _urlopen(str(filepath_or_buffer))
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 564, in error
    result = self._call_chain(*args)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Users\ritvik\AppData\Local\Continuum\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

[Tim Cera]

Thank you for finding this bug. For a quick work around the dates have to include the hour, for example '2016-01-01T00'. This is something that I should do in the background. Also, the ending date would give a problem and I am not catching that and giving a reasonable error message.

There also is another problem, you have to include the full name of the variable you want.

This should work for soil moisture in the top 40 cm.

from tsgettoolbox import tsgettoolbox
df = tsgettoolbox.ldas(lat=35.86,
                        lon=104.2,
                        variable='GLDAS:GLDAS_NOAH025_3H.001:SOILM10-40cm',
                        startDate='2016-01-01T00',
                        endDate='2016-12-01T00')

[Ritvik Sahajpal]

Thanks Tim! This is very helpful. Thanks again for an excellent package

[Tim Cera]

The latest version 7.13.5.8 adds a very flexible date parser and reformats the date as needed for the web service in the background.

Still haven't put together an error for the situation where you try to get data from NLDAS that isn't there.

[Tim Cera]

Closing this in favor of splitting the problems into issue #3 and issue #4