1. Matt Chaput
  2. whoosh
  3. Issues
Issue #311 invalid

LockError when using Moin2 Global Index to upload multiple files on Windows 7

created an issue

When uploading 2 or more 1k files this error occurs 100% of the time (Windows 7, MoinMoin 2 built-in server):

    2013-02-11 07:44:24,066 INFO werkzeug:116 "POST /%2Bjfu-server HTTP/1.1" 500 -
    Traceback (most recent call last):
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1701, in __call__
        return self.wsgi_app(environ, start_response)
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1689, in wsgi_app
        response = self.make_response(self.handle_exception(e))
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1687, in wsgi_app
        response = self.full_dispatch_request()
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1360, in full_dispatch_request
        rv = self.handle_user_exception(e)
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1358, in full_dispatch_request
        rv = self.dispatch_request()
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\flask\app.py", line 1344, in dispatch_request
        return self.view_functions[rule.endpoint](**req.view_args)
      File "c:\bitbucket\m2-x4\MoinMoin\apps\frontend\views.py", line 718, in jfu_server
        revid, size = item.modify({}, data, contenttype_guessed=contenttype)
      File "c:\bitbucket\m2-x4\MoinMoin\items\__init__.py", line 372, in modify
        return self._save(meta, data, contenttype_guessed=contenttype_guessed, comment=comment)
      File "c:\bitbucket\m2-x4\MoinMoin\items\__init__.py", line 468, in _save
      File "c:\bitbucket\m2-x4\MoinMoin\storage\middleware\protecting.py", line 299, in store_revision
        rev = self.item.store_revision(meta, data, overwrite=overwrite, return_rev=return_rev, **kw)
      File "c:\bitbucket\m2-x4\MoinMoin\storage\middleware\indexing.py", line 1071, in store_revision
        self.indexer.index_revision(meta, content, backend_name)
      File "c:\bitbucket\m2-x4\MoinMoin\storage\middleware\indexing.py", line 455, in index_revision
        writer = self.ix[ALL_REVS].writer()
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\whoosh\filedb\fileindex.py", line 258, in writer
        return SegmentWriter(self, **kwargs)
      File "C:\Bitbucket\m2-x4\env\lib\site-packages\whoosh\filedb\filewriting.py", line 137, in __init__
        raise LockError

If many files are uploaded then the server log shows several errors. If 4 1k files are uploaded, there are 2 tracebacks; 10 1k files yield 5 tracebacks.

I believe the problem to be related to the timeout parameter within whoosh/filedb/filewriting.py near line 128.

    class SegmentWriter(IndexWriter):
        def __init__(self, ix, poolclass=None, timeout=0.0, delay=0.1, _lk=True,
                     limitmb=128, docbase=0, codec=None, compound=True, **kwargs):

Setting the timeout delay to 1.0 and uploading 20 2+MB files yielded one traceback. Setting the timeout delay to 5.0 and uploading 36 2+MB files resulted in no failures.

I do not have a remote Windows server available to test the effect of a slow uplink. The problem cannot be reproduced on a Linux PC or a remote Linux server.

Comments (4)

  1. Matt Chaput repo owner

    Nope, it's a classic single writer, multiple reader system. If you have simultaneous incoming writes, you need to layer on a queue and/or lock to serialize them into single writes.

  2. Log in to comment