runname = writer.pool.runs[0] IndexError: list index out of range in multiproc.py line 54

Brook Humphrey avatarBrook Humphrey created an issue

I'm not sure why I get this. I'm using whoosh as a backend for indexing epubs. Code id already in place and if I index each page in the epub one at a time then it works no issue but when I do a split on the lines so that it can be broken down by paragraph is when I get this.

It almost seems like the queue is expecting more threads than are there?

I'd be willing to help with this but I'm not really sure were to start as I have not dealt much with multiproc yet.

Process SubWriterTask-3: Traceback (most recent call last): File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap self.run() File "C:\Users\Brook_Humphrey\My Documents\Aptana Studio 3 Workspace\beams_of_light-webview\src\lib\whoosh\multiproc.py", line 125, in run runname, segment = finish_subsegment(writer, k) File "C:\Users\Brook_Humphrey\My Documents\Aptana Studio 3 Workspace\beams_of_light-webview\src\lib\whoosh\multiproc.py", line 49, in finish_subsegment runname = writer.pool.runs[0] IndexError: list index out of range

Comments (4)

  1. Brook Humphrey

    I'm still not sure why I get this but I created instead a BufferedWriter since I'm indexing a bunch of smaller documents and the buffered writer does not show this issue.

    Thanks

  2. Stephane Boisson

    I have the same error when I commit a bath index job.

    I disabled multiproc as a workaround to avoid the bug.

    Process SubWriterTask-2:
    Traceback (most recent call last):
      File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
        self.run()
      File "/usr/lib/python2.6/site-packages/whoosh/multiproc.py", line 128, in run
        runname, fieldnames, segment = finish_subsegment(writer, k)
      File "/usr/lib/python2.6/site-packages/whoosh/multiproc.py", line 49, in finish_subsegment
        runname = writer.pool.runs[0]
    IndexError: list index out of range
    Traceback (most recent call last):
    (...)
      File "amazintv/processors/indexer.py", line 189, in finish
        self.tvseries_writer.commit(optimize=True)
      File "/usr/lib/python2.6/site-packages/whoosh/multiproc.py", line 252, in commit
        self._commit(mergetype, optimize, merge)
      File "/usr/lib/python2.6/site-packages/whoosh/multiproc.py", line 277, in _commit
        results.append(self.resultqueue.get(timeout=5))
      File "/usr/lib64/python2.6/multiprocessing/queues.py", line 104, in get
        raise Empty
    Queue.Empty
    
  3. Matt Chaput

    This seems to be a heisenbug in the multiprocessing module. It shouldn't be possible for the queue to be empty. I created some test code but it works every time except once when it showed the error above. :(

  4. Sai Manoj Kumar Yadlapati

    I get the same error while indexing using multiproc,

    Process SubWriterTask-15:
    Traceback (most recent call last):
    Process SubWriterTask-16:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 128, in run
        self.run()
        runname, fieldnames, segment = finish_subsegment(writer, k)
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 49, in finish_subsegment
        runname = writer.pool.runs[0]
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 128, in run
        runname, fieldnames, segment = finish_subsegment(writer, k)
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 49, in finish_subsegment
        runname = writer.pool.runs[0]
    IndexError: list index out of range
    IndexError: list index out of range
    Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/usr/local/lib/python2.7/dist-packages/memory_profiler.py", line 821, in <module>
        execfile(__file__, ns, ns)
      File "lda2.py", line 34, in <module>
        exp.run_experiment()
      File "lda2.py", line 26, in run_experiment
        self.index_dataset(sel_terms)
      File "experiment.py", line 32, in index_dataset
        writer.commit(optimize=True)
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 252, in commit
        self._commit(mergetype, optimize, merge)
      File "/usr/local/lib/python2.7/dist-packages/whoosh/multiproc.py", line 277, in _commit
        results.append(self.resultqueue.get(timeout=5))
      File "/usr/lib/python2.7/multiprocessing/queues.py", line 132, in get
        raise Empty
    Queue.Empty
    

    version of multiprocessing I am using is '0.70a1', whoosh version is 2,5,6. Also I am using multiprocessing in my own code, don't know whether it is interfering with it. I am using multiprocessing.Pool.

  5. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.