Issue #314 new

PicklingError: Can't pickle <type 'function'>, Apache, Windows 7

Roger Haase
created an issue

On my Windows 7 test system there are multiple WSGI applications running under Apache on Windows 7. This includes Moin2, Moin 1.9.6, a werkzeug app serving images, and a old Webware for Python app.

If the Apache server is restarted and any application other than Moin2 is accessed first, then Moin2 is accessed to modify and save an item, then the result is a Pickling error within env\Lib\site-packages\whoosh\filedb\ This failure happens 100% of the time.

 mod_wsgi (pid=3936): Exception occurred processing WSGI script 'C:/home/web/m2-fixedleft/moin2.wsgi'., referer:
Traceback (most recent call last):, referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1701, in __call__, referer:
    return self.wsgi_app(environ, start_response), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1689, in wsgi_app, referer:
    response = self.make_response(self.handle_exception(e)), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1687, in wsgi_app, referer:
    response = self.full_dispatch_request(), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1360, in full_dispatch_request, referer:
    rv = self.handle_user_exception(e), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1358, in full_dispatch_request, referer:
    rv = self.dispatch_request(), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\", line 1344, in dispatch_request, referer:
    return self.view_functions[rule.endpoint](**req.view_args), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\apps\\frontend\\", line 493, in modify_item, referer:
    return item.do_modify(), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\", line 727, in do_modify, referer:
    self.modify(meta, data, comment, contenttype_guessed, contenttype_qs), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\", line 395, in modify, referer:
    return self._save(meta, data, contenttype_guessed=contenttype_guessed, comment=comment), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\", line 511, in _save, referer:
    return_rev=True,, referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\", line 300, in store_revision, referer:
    rev = self.item.store_revision(meta, data, overwrite=overwrite, return_rev=return_rev, **kw), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\", line 1077, in store_revision, referer:
    self.indexer.index_revision(meta, content, backend_name), referer:
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\", line 458, in index_revision, referer:
    writer.update_document(**doc)  # update, because store_revision() may give us an existing revid, referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\", line 80, in __exit__, referer:
    self.commit(), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\", line 518, in commit, referer:
    self._commit_toc(finalsegments), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\", line 454, in _commit_toc, referer:
    segments, self.generation), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\codec\\", line 123, in commit_toc, referer:
    toc.write(storage, indexname), referer:
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\", line 161, in write, referer:
    stream.write_string(pickle.dumps(schema, -1)), referer:
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed, referer:

This same failure was happening on CentOS, but the problem went away after adding a WSGIDaemonProcess for each app. Similar to:

WSGIDaemonProcess moin2_wsgi processes=1 threads=3 stack-size=524288
WSGIScriptAlias /moin2/ /home/rockart/webapps/web/moin-2.0/moin2.wsgi/
<Location /moin2>
    WSGIProcessGroup moin2_wsgi

Unfortunately, Windows Apache does not support WSGIDaemonProcess.

One easy workaround is to always access Moin2 first after every Apache restart. It is not necessary to modify and save an item.

Another workaround that was discovered is to modify whoosh/filedb/ near line 32 as follows:

    #~ from whoosh.compat import pickle, string_type, xrange
    from whoosh.compat import string_type, xrange
    import pickle  # workaround PicklingError

... which replaces the cPickle module imported in whoosh.comat with the pickle module. (importing cPickle as pickle here fails to prevent the error).

Comments (7)

  1. Wim Lewis

    I've been hitting this problem, or one similar to it, with Whoosh 2.5.6 in a Django+WSGI application (MacOSX, python 2.7.5):

     File ".../python2.7/site-packages/django/contrib/admin/", line 1084, in response_action
       response = func(self, request, queryset)
     File ".../", line 69, in re_index
     File ".../python2.7/site-packages/whoosh/", line 935, in commit
     File ".../python2.7/site-packages/whoosh/", line 882, in _commit_toc
       toc.write(, self.indexname)
     File ".../python2.7/site-packages/whoosh/", line 693, in write
     File ".../python2.7/", line 70, in _reduce_ex
       raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle function objects

    (I've trimmed the paths a bit for readability's sake.)

  2. Wim Lewis

    This is the schema:

        freetext_analyzer = StemmingAnalyzer() | CharsetFilter(accent_map)
        schema = Schema(path = ID(stored=True, unique=True),
                        file_mtime = DATETIME(stored=True),
                        name = TEXT(stored=False, field_boost=2.0),
                        description = TEXT(stored=False, field_boost=1.5, analyzer=freetext_analyzer),
                        content = TEXT(analyzer=freetext_analyzer))

    I've added TypeError and ValueError to that except block in whoosh/, so next time it happens I'll at least know which field it is. (Like Roger Haase, I usually don't have the problem, just sometimes. I haven't found an easy way to force it to happen.)

  3. manelclos

    I have the same problem on Linux, Python 2.7.3 and Whoosh 2.6.0. Simple Schema will fail:

    schema = Schema(name=TEXT())

    can't pickle function objects, here:

    #virtualenv_path/.../site-packages/whoosh/ in write, line 693
                stream.write_string(pickle.dumps(schema, -1))
            except pickle.PicklingError:
                # Try to narrow down the error to a single field
                for fieldname, field in schema.items():
    **                    pickle.dumps(field)**
                    except pickle.PicklingError:
                        e = sys.exc_info()[1]
                        raise pickle.PicklingError("%s %s=%r" % (e, fieldname, field))
                # Otherwise, re-raise the original exception
  4. Log in to comment