1. Matt Chaput
  2. whoosh
Issue #309 resolved

mmap.error: [Errno 12] Cannot allocate memory

Koen Crolla
created an issue

While adding a particularly large document to my index I got the following error:

Traceback (most recent call last):
  File "./spider.py", line 238, in <module>
    html=expand_urls(post[u'com']))
  File "/usr/local/lib/python2.6/dist-packages/whoosh/writing.py", line 350, in update_document
    with self.searcher() as s:
  File "/usr/local/lib/python2.6/dist-packages/whoosh/writing.py", line 167, in searcher
    return Searcher(self.reader(), **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filewriting.py", line 241, in reader
    self.generation, reuse=reuse)
  File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 321, in _reader
    readers = [segreader(segment) for segment in segments]
  File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 310, in segreader
    generation=generation)
  File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filereading.py", line 70, in __init__
    self.files = OverlayStorage(segment.open_compound_file(storage),
  File "/usr/local/lib/python2.6/dist-packages/whoosh/codec/base.py", line 656, in open_compound_file
    return CompoundStorage(storage, name)
  File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/compound.py", line 58, in __init__
    self.source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ)
mmap.error: [Errno 12] Cannot allocate memory

The error happens inside a try/except block that explicitly ignores errno.ENOMEM, which Errno 12 is; the problem is that the except clause catches OSError instead of mmap.error.

Apparently mmap.mmap mostly raises WindowsError (a subclass of OSError) on Windows but mmap.error on Unix. The except clause should probably catch both mmap.error and OSError (WindowsError itself doesn't exist on Unix).

Comments (3)

  1. Thomas Waldmann

    patch looks ok. did you reproduce the issue and test your patch on Windows as well as on POSIX (Linux)?

    the only question is if we really should catch OSError or rather WindowsError. Also, some comment there about windows vs. posix would be nice, especially as python mmap docs are somehow lacking infos about exceptions.

    note: needs to get fixed in 2.4x branch as well as in default ("2.5") branch.

  2. Koen Crolla reporter

    Yep, I did test it on Windows as well; mmap raises a WindowsError (with winerror code 8) there if it doesn't have enough memory, and catching (mmap.error, OSError) works as expected.

    WindowsError doesn't exist on platforms other than Windows at all (something that isn't obvious from the Python docs either), so the only option is catching either OSError, its parent, or EnvironmentError, OSError's parent. Fortunately both have a meaningful errno attribute even on Windows.

  3. Log in to comment