Issue #298 resolved

charset_table_to_dict - pickle error‏

rutokusu
created an issue

Hello, it seems that when using whoosh.analysis.CharsetFilter with charset.charset_table_to_dict(charset.default_charset), I receive PicklingError.

Code sample:

import whoosh.analysis
from whoosh.fields import Schema, TEXT, ID
from whoosh.support import charset
import whoosh.index

charmap = charset.charset_table_to_dict(charset.default_charset)
analyzer = whoosh.analysis.StandardAnalyzer() | whoosh.analysis.CharsetFilter(charmap)
schema = Schema(
    content=TEXT(analyzer=analyzer)
)
index = whoosh.index.create_in('index', schema)

Error traceback:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    index = whoosh.index.create_in('index', schema)
  File "build\bdist.win32\egg\whoosh\index.py", line 91, in create_in
  File "build\bdist.win32\egg\whoosh\filedb\filestore.py", line 73, in create_index
  File "build\bdist.win32\egg\whoosh\filedb\filestore.py", line 47, in create_index
  File "build\bdist.win32\egg\whoosh\filedb\fileindex.py", line 83, in create
  File "build\bdist.win32\egg\whoosh\filedb\fileindex.py", line 161, in write
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I'm using Whoosh 2.4.1 on Python 2.7 x32 (Windows).

I believe the problem lies with the fact that charset_table_to_dict returns defaultdict and this type of object cannot be serialized by pickle. I have confirmed this by casting charmap to dict, after which it worked (no more pickle errors). I'm not really sure this is correct solution (at least as a workaround) and whether or not it will break something.

Comments (1)

  1. Log in to comment