netcdf cftime update needed for demodata / mswms

[Reimar Bauer]

Fresh miniconda3 installation, everything removed beforehand

A downgrade to 1.4.0 fixed it

conda install netcdf4=1.4.0

netcdf4: 1.4.1-py36h4b4f87f_0 --> 1.4.0-py36ha06eab4_1

$ demodata
To use this setup you need the mss_wms_settings.py in your python path e.g. 
export PYTHONPATH=~/mss
(mssenv) user@host:~/mss$ mswms
2018-09-02 20:31:14               setup || Files identified for domain 'EUR_LL015': ['20121017_12_ecmwf_forecast.CC.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.SEA.EUR_LL015.036.sfc.nc', '20121017_12_ecmwf_forecast.PRESSURE_LEVELS.EUR_LL015.036.pl.nc', '20121017_12_ecmwf_forecast.PV_derived.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.CIWC.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.Q.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.ProbWCB_LAGRANTO_derived.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.P_derived.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.U.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.V.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.CLWC.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.ProbWCB_LAGRANTO_derived.EUR_LL015.036.sfc.nc', '20121017_12_ecmwf_forecast.SFC.EUR_LL015.036.sfc.nc', '20121017_12_ecmwf_forecast.W.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.ALTITUDE_LEVELS.EUR_LL015.036.al.nc', '20121017_12_ecmwf_forecast.T.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.PVU.EUR_LL015.036.pv.nc', '20121017_12_ecmwf_forecast.EMAC.EUR_LL015.036.ml.nc', '20121017_12_ecmwf_forecast.THETA_LEVELS.EUR_LL015.036.tl.nc']
2018-09-02 20:31:14               setup || Opening candidate '20121017_12_ecmwf_forecast.CC.EUR_LL015.036.ml.nc'
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/mssenv/bin/mswms", line 7, in <module>
    from mslib.mswms.mswms import main
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/mslib/mswms/mswms.py", line 39, in <module>
    from mslib.mswms.wms import mss_wms_settings, mss_wms_auth
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/mslib/mswms/wms.py", line 510, in <module>
    app = WMSServer()
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/mslib/mswms/wms.py", line 120, in __init__
    data_access_dict[key].setup()
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/mslib/mswms/dataaccess.py", line 180, in setup
    valid_times = netCDF4tools.num2date(time_var[:], time_var.units)
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/mslib/netCDF4tools.py", line 145, in num2date
    return netCDF4.num2date(times, units, calendar=calendar)
  File "cftime/_cftime.pyx", line 290, in cftime._cftime.num2date
  File "cftime/_cftime.pyx", line 1211, in cftime._cftime.utime.num2date
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 279, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/home/user/miniconda3/envs/mssenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: cannot reshape array of size 1 into shape (7,)

[Reimar Bauer]

filed this also to https://github.com/conda-forge/netcdf4-feedstock/issues/58

[Joern Ungermann]

Occurs also on Windows.

This is a minimal testcase demonstrating the problem:

import numpy as np
import netCDF4 as nc

a = np.asarray([0, 6, 12])
print(nc.num2date(a, "hours since 2012-10-17T12:00:00.000Z"))  # <- fine
b = np.ma.masked_invalid(a)
print(type(b), b.shape, b.mask)
print(nc.num2date(b, "hours since 2012-10-17T12:00:00.000Z"))  # <- fine
b = np.ma.asarray(a)
print(type(b), b.shape, b.mask)
print(nc.num2date(b, "hours since 2012-10-17T12:00:00.000Z"))  # <- Exception

Output:

[datetime.datetime(2012, 10, 17, 12, 0)
 datetime.datetime(2012, 10, 17, 18, 0)
 datetime.datetime(2012, 10, 18, 0, 0)]
<class 'numpy.ma.core.MaskedArray'> (3,) [False False False]
[datetime.datetime(2012, 10, 17, 12, 0)
 datetime.datetime(2012, 10, 17, 18, 0)
 datetime.datetime(2012, 10, 18, 0, 0)]
<class 'numpy.ma.core.MaskedArray'> (3,) False
Traceback (most recent call last):
  File "b.py", line 11, in <module>
    print(nc.num2date(b, "hours since 2012-10-17T12:00:00.000Z"))  # <- Exception
  File "cftime/_cftime.pyx", line 290, in cftime._cftime.num2date
  File "cftime/_cftime.pyx", line 1211, in cftime._cftime.utime.num2date
  File "C:\Users\icg173\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py", line 257, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "C:\Users\icg173\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py", line 52, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: cannot reshape array of size 1 into shape (3,)

The problem seemingly occurs, when a masked array is supplied that uses only a single boolean (np.ma.nomask) instead of a full boolean array.

[Joern Ungermann]

If we patch our netCDF4 module to

def num2date(times, units, calendar='standard'):
    """
    Extension to the netCDF4.num2date() function to correctly handle
    time strings of format '2010-01-01T00:00:00Z', as used by netcdf-java.

    Refer to netCDF4.num2date() for further documentation.
    """
    import numpy as np
    import numpy as np
    if hasattr(times, "mask") and times.mask is np.ma.nomask:
        times = times.data
    return netCDF4.num2date(times, units, calendar=calendar)

we ensure that the masked array has either a full mask or that a normal array is supplied, neither of which give problems.

[Reimar Bauer]

@joernu76 please do so. We should have later a test which is now disabled for this behaviour.

[Reimar Bauer]

the newer netCDF4 module needs a newer cftime, adjusted dependency, fixes #366

→ <<cset 975bbbabfbbb>>

[Reimar Bauer]

Merged in ReimarBauer/mss/stable (pull request #584)

the newer netCDF4 module needs a newer cftime, adjusted dependency, fixes #366

→ <<cset 6692a8bfe62e>>

[Reimar Bauer]

https://github.com/conda-forge/netcdf4-feedstock/issues/58