insert doesn't work correctly when mask is not an array but a boolean

Issue #22 new
Femto Trader
created an issue

Hello,

I'm working with Monary and xray and faced a problem because xray.DataArray.to_masked_array doesn't return an array of boolean but just a boolean.

In [1]: ma.masked_array(df['Bid'].values, df['Bid'].isnull())
Out[1]:
masked_array(data = [0.81 0.88805 0.8880899999999999 ..., 0.87531 0.87531 0.87531],
             mask = [False False False ..., False False False],
       fill_value = 1e+20)

In [2]: ds.Bid.to_masked_array()
Out[2]:
masked_array(data = [ 0.81     0.88805  0.88809 ...,  0.87531  0.87531  0.87531],
             mask = False,
       fill_value = 1e+20)

So we have

mask = False,

instead of

 mask = [False False False ..., False False False],

Here is my code With sample data from https://drive.google.com/file/d/0B8iUtWjZOTqla3ZZTC1FS0pkZXc/view?usp=sharing

import pandas as pd
import xray
df = pd.read_csv("AUDUSD-2014-01.csv", names=['Symbol', 'Date', 'Bid', 'Ask'])
df['Date'] = pd.to_datetime(df['Date'], format='%Y%m%d %H:%M:%S.%f')
ds = xray.Dataset.from_dataframe(df)
ds = ds[['Bid', 'Ask']]
lst_cols = list(map(lambda col: ds[col].to_masked_array(), ds.data_vars))
mparams = monary.MonaryParam.from_lists(lst_cols, list(ds.data_vars), ['float64', 'float64'])
m.insert('monary_db', 'ticks', mparams)

It raises

//anaconda/lib/python3.4/site-packages/monary/monary.py in insert(self, db, coll, params, write_concern)
    763             for i, param in enumerate(params):
    764                 data_p = param.array.data.ctypes.data_as(ctypes.c_void_p)
--> 765                 mask_p = param.array.mask.ctypes.data_as(ctypes.c_void_p)
    766
    767                 if cmonary.monary_set_column_item(

AttributeError: 'numpy.bool_' object has no attribute 'ctypes'

see lst_cols

[masked_array(data = [ 0.88796  0.88805  0.88809 ...,  0.87531  0.87531  0.87531],
              mask = False,
        fill_value = 1e+20),
 masked_array(data = [ 0.88922  0.88914  0.8891  ...,  0.87588  0.87574  0.87574],
              mask = False,
        fill_value = 1e+20)]

Maybe a try / except could fix this

try:
    mask_p = param.array.mask.ctypes.data_as(ctypes.c_void_p)
except AttributeError:
    mask_p = False

An other option could be on xray side to_masked_array could have a parameter to always return array https://github.com/xray/xray/issues/684

Kind regards

Comments (3)

  1. Log in to comment