OSError: [Errno 22] Invalid argument when optimizing the index

Issue #475 new
mx2048
created an issue

Hello, I was trying to optimize my index containing several segments of total size about 32 GB using the following code:

def optimize_index(folder):
    ix = index.open_dir(folder)
    writer = ix.writer(limitmb=1024)
    writer.commit(optimize=True)

And when the size of the MAIN.tmp folder containing *.run files was 137 GB, and when, in the main directory, all *.col files corresponding to text fields were generated, I got an error.

Traceback (most recent call last):
  File "whoosh_index.py", line 97, in <module>
    optimize_index(folder)
  File "whoosh_index.py", line 72, in optimize_index
    writer.commit(optimize=True)
  File "C:\Program Files\Python3\lib\site-packages\whoosh\writing.py", line 927, in commit
    finalsegments.append(self._finalize_segment())
  File "C:\Program Files\Python3\lib\site-packages\whoosh\writing.py", line 866, in _finalize_segment
    self._flush_segment()
  File "C:\Program Files\Python3\lib\site-packages\whoosh\writing.py", line 835, in _flush_segment
    postings = self.pool.iter_postings()
  File "C:\Program Files\Python3\lib\site-packages\whoosh\writing.py", line 176, in iter_postings
    return self.items()
  File "C:\Program Files\Python3\lib\site-packages\whoosh\externalsort.py", line 220, in items
    self.reduce_to(maxfiles, maxfiles)
  File "C:\Program Files\Python3\lib\site-packages\whoosh\externalsort.py", line 199, in reduce_to
    self._write_run(f, self._merge_runs(tomerge))
  File "C:\Program Files\Python3\lib\site-packages\whoosh\externalsort.py", line 161, in _write_run
    for item in items:
  File "C:\Program Files\Python3\lib\site-packages\whoosh\externalsort.py", line 149, in _merge_runs
    for item in imerge(iters):
  File "C:\Program Files\Python3\lib\heapq.py", line 359, in merge
    s[0] = next()           # raises StopIteration when exhausted
  File "C:\Program Files\Python3\lib\site-packages\whoosh\externalsort.py", line 140, in _read_run
    yield load(f)
OSError: [Errno 22] Invalid argument

I used Python 3.6.4

Any suggestions are really appreciated!

Comments (0)

  1. Log in to comment