Issue #314 new

PicklingError: Can't pickle <type 'function'>, Apache, Windows 7

Roger Haase
created an issue

On my Windows 7 test system there are multiple WSGI applications running under Apache on Windows 7. This includes Moin2, Moin 1.9.6, a werkzeug app serving images, and a old Webware for Python app.

If the Apache server is restarted and any application other than Moin2 is accessed first, then Moin2 is accessed to modify and save an item, then the result is a Pickling error within env\Lib\site-packages\whoosh\filedb\fileindex.py. This failure happens 100% of the time.

 mod_wsgi (pid=3936): Exception occurred processing WSGI script 'C:/home/web/m2-fixedleft/moin2.wsgi'., referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
Traceback (most recent call last):, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1701, in __call__, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    return self.wsgi_app(environ, start_response), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1689, in wsgi_app, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    response = self.make_response(self.handle_exception(e)), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1687, in wsgi_app, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    response = self.full_dispatch_request(), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1360, in full_dispatch_request, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    rv = self.handle_user_exception(e), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1358, in full_dispatch_request, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    rv = self.dispatch_request(), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\flask\\app.py", line 1344, in dispatch_request, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    return self.view_functions[rule.endpoint](**req.view_args), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\apps\\frontend\\views.py", line 493, in modify_item, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    return item.do_modify(), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\__init__.py", line 727, in do_modify, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    self.modify(meta, data, comment, contenttype_guessed, contenttype_qs), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\__init__.py", line 395, in modify, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    return self._save(meta, data, contenttype_guessed=contenttype_guessed, comment=comment), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\items\\__init__.py", line 511, in _save, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    return_rev=True,, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\protecting.py", line 300, in store_revision, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    rev = self.item.store_revision(meta, data, overwrite=overwrite, return_rev=return_rev, **kw), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\indexing.py", line 1077, in store_revision, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    self.indexer.index_revision(meta, content, backend_name), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\MoinMoin\\storage\\middleware\\indexing.py", line 458, in index_revision, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    writer.update_document(**doc)  # update, because store_revision() may give us an existing revid, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\writing.py", line 80, in __exit__, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    self.commit(), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\filewriting.py", line 518, in commit, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    self._commit_toc(finalsegments), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\filewriting.py", line 454, in _commit_toc, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    segments, self.generation), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\codec\\whoosh2.py", line 123, in commit_toc, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    toc.write(storage, indexname), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
  File "C:\\home\\web\\m2-fixedleft\\env\\Lib\\site-packages\\whoosh\\filedb\\fileindex.py", line 161, in write, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
    stream.write_string(pickle.dumps(schema, -1)), referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed, referer: http://antelope.digitalrockart.org/moin2/%2Bmodify/FrontPage

This same failure was happening on CentOS, but the problem went away after adding a WSGIDaemonProcess for each app. Similar to:

WSGIDaemonProcess moin2_wsgi processes=1 threads=3 stack-size=524288
WSGIScriptAlias /moin2/ /home/rockart/webapps/web/moin-2.0/moin2.wsgi/
<Location /moin2>
    WSGIProcessGroup moin2_wsgi
</Location>

Unfortunately, Windows Apache does not support WSGIDaemonProcess.

One easy workaround is to always access Moin2 first after every Apache restart. It is not necessary to modify and save an item.

Another workaround that was discovered is to modify whoosh/filedb/fileindex.py near line 32 as follows:

    #~ from whoosh.compat import pickle, string_type, xrange
    from whoosh.compat import string_type, xrange
    import pickle  # workaround PicklingError

... which replaces the cPickle module imported in whoosh.comat with the pickle module. (importing cPickle as pickle here fails to prevent the error).

Comments (7)

  1. Wim Lewis

    I've been hitting this problem, or one similar to it, with Whoosh 2.5.6 in a Django+WSGI application (MacOSX, python 2.7.5):

     File ".../python2.7/site-packages/django/contrib/admin/options.py", line 1084, in response_action
       response = func(self, request, queryset)
    
     File ".../admin.py", line 69, in re_index
       writer.commit()
    
     File ".../python2.7/site-packages/whoosh/writing.py", line 935, in commit
       self._commit_toc(finalsegments)
    
     File ".../python2.7/site-packages/whoosh/writing.py", line 882, in _commit_toc
       toc.write(self.storage, self.indexname)
    
     File ".../python2.7/site-packages/whoosh/index.py", line 693, in write
       pickle.dumps(field)
    
     File ".../python2.7/copy_reg.py", line 70, in _reduce_ex
       raise TypeError, "can't pickle %s objects" % base.__name__
    
    TypeError: can't pickle function objects
    

    (I've trimmed the paths a bit for readability's sake.)

  2. Wim Lewis

    This is the schema:

        freetext_analyzer = StemmingAnalyzer() | CharsetFilter(accent_map)
    
        schema = Schema(path = ID(stored=True, unique=True),
                        file_mtime = DATETIME(stored=True),
                        name = TEXT(stored=False, field_boost=2.0),
                        description = TEXT(stored=False, field_boost=1.5, analyzer=freetext_analyzer),
                        content = TEXT(analyzer=freetext_analyzer))
    

    I've added TypeError and ValueError to that except block in whoosh/index.py, so next time it happens I'll at least know which field it is. (Like Roger Haase, I usually don't have the problem, just sometimes. I haven't found an easy way to force it to happen.)

  3. manelclos

    I have the same problem on Linux, Python 2.7.3 and Whoosh 2.6.0. Simple Schema will fail:

    schema = Schema(name=TEXT())

    can't pickle function objects, here:

    #virtualenv_path/.../site-packages/whoosh/index.py in write, line 693
            try:
                stream.write_string(pickle.dumps(schema, -1))
            except pickle.PicklingError:
                # Try to narrow down the error to a single field
                for fieldname, field in schema.items():
                    try:
    **                    pickle.dumps(field)**
                    except pickle.PicklingError:
                        e = sys.exc_info()[1]
                        raise pickle.PicklingError("%s %s=%r" % (e, fieldname, field))
                # Otherwise, re-raise the original exception
                raise
    
  4. Log in to comment