whoosh index tmp files going to /tmp

Issue #885 new
Anonymous created an issue

We've noticed that the reindexing is creating several files in the /tmp directory on Linux. below is a conversation I had on IRC with marcinkuzminski. Submitting this to be able to control which directory the whoosh indexing temp files are temporarily stored.

[18:07] <cdsmmeyer> hello.  When I do reindexing I see several files like this in the /tmp directory on linux.  what are they and how can I redirect them to something other than /tmp?
[18:07] <cdsmmeyer> -rw------- 1 rhodecode rhodecode 57481268 Aug  1 10:47 whoosh_HG_INDEX_jwmHIS.run
[18:07] <marcinkuzminski> ahh i didn't have time to answer that on discussion group
[18:08] <marcinkuzminski> cdsmmeyer: it's whoosh temporary files created during indexing
[18:08] <cdsmmeyer> do they get deleted when indexing is done?
[18:08] <marcinkuzminski> they should be gone when indexing is finished and index commited
[18:08] <cdsmmeyer> is there anyway to have them created somewhere else other than /tmp?
[18:09] <marcinkuzminski> I don't know i would have to check in whoosh docs
[18:09] <cdsmmeyer> ok.  so far we love the features of rhodecode :)
[18:10] <marcinkuzminski> great :)
[18:10] <cdsmmeyer> keep up the good work
[18:10] <marcinkuzminski> did you seen our new UI at secure.rhodecode.org and website rhodecode.com ?
[18:11] <marcinkuzminski> btw: https://groups.google.com/forum/#!topic/whoosh/4av9dOUXIPc
[18:11] <cdsmmeyer> i have not yet.  big changes on the way?
[18:11] <marcinkuzminski> yeah, we now started to work full time on it ;)
[18:11] <cdsmmeyer> cool.   wow that topic was created in 2010.  no replies huh?
[18:13] <marcinkuzminski> seems to be fixed
[18:13] <marcinkuzminski> https://bitbucket.org/mchaput/whoosh/issue/48/temp-directories-are-not-deleted-when
[18:14] <marcinkuzminski> from the commit it seems that this is configurable
[18:15] <cdsmmeyer> the location of the temp files?
[18:15] <marcinkuzminski> yes
[18:16] <marcinkuzminski> there is a 'dir=/path' param that can be passed in into a whoosh writer
[18:17] <cdsmmeyer> so not able to put in the production.ini file I take it?
[18:17] <marcinkuzminski> no
[18:17] <marcinkuzminski> we would have to add it

Comments (6)

  1. valentijnscholten

    I think the main problem is that the index files don't get deleted after indexing is complete. Over the weekend our rhodecode server went out of diskspace because of >10GB of whoosh index files being left behind in /tmp.(rhodecode 1.7.1).

  2. valentijnscholten

    I think it finishes succesfully:

    2013-10-12 07:07:27.982 DEBUG [whoosh_indexer] >> COMMITING CHANGES TO FILE INDEX <<

    No more logging after that until the next index run.

    I see a ticket about tmp files in the whoosh tracker, but I am not sure if that's talking about temp dirs only or also temp files. In my opinion whoosh should remove the files when done.


    Maybe an upgrade to newest whoosh?

  3. Log in to comment