1. Marcin Kuzminski
  2. RhodeCode
  3. Issues

Issues

Issue #885 new

whoosh index tmp files going to /tmp

Anonymous created an issue

We've noticed that the reindexing is creating several files in the /tmp directory on Linux. below is a conversation I had on IRC with marcinkuzminski. Submitting this to be able to control which directory the whoosh indexing temp files are temporarily stored.

[18:07] <cdsmmeyer> hello.  When I do reindexing I see several files like this in the /tmp directory on linux.  what are they and how can I redirect them to something other than /tmp?
[18:07] <cdsmmeyer> -rw------- 1 rhodecode rhodecode 57481268 Aug  1 10:47 whoosh_HG_INDEX_jwmHIS.run
[18:07] <marcinkuzminski> ahh i didn't have time to answer that on discussion group
[18:08] <marcinkuzminski> cdsmmeyer: it's whoosh temporary files created during indexing
[18:08] <cdsmmeyer> do they get deleted when indexing is done?
[18:08] <marcinkuzminski> they should be gone when indexing is finished and index commited
[18:08] <cdsmmeyer> is there anyway to have them created somewhere else other than /tmp?
[18:09] <marcinkuzminski> I don't know i would have to check in whoosh docs
[18:09] <cdsmmeyer> ok.  so far we love the features of rhodecode :)
[18:10] <marcinkuzminski> great :)
[18:10] <cdsmmeyer> keep up the good work
[18:10] <marcinkuzminski> did you seen our new UI at secure.rhodecode.org and website rhodecode.com ?
[18:11] <marcinkuzminski> btw: https://groups.google.com/forum/#!topic/whoosh/4av9dOUXIPc
[18:11] <cdsmmeyer> i have not yet.  big changes on the way?
[18:11] <marcinkuzminski> yeah, we now started to work full time on it ;)
[18:11] <cdsmmeyer> cool.   wow that topic was created in 2010.  no replies huh?
[18:13] <marcinkuzminski> seems to be fixed
[18:13] <marcinkuzminski> https://bitbucket.org/mchaput/whoosh/issue/48/temp-directories-are-not-deleted-when
[18:14] <marcinkuzminski> from the commit it seems that this is configurable
[18:15] <cdsmmeyer> the location of the temp files?
[18:15] <marcinkuzminski> yes
[18:16] <marcinkuzminski> there is a 'dir=/path' param that can be passed in into a whoosh writer
[18:17] <cdsmmeyer> so not able to put in the production.ini file I take it?
[18:17] <marcinkuzminski> no
[18:17] <marcinkuzminski> we would have to add it

Comments (6)

  1. valentijnscholten

    I think the main problem is that the index files don't get deleted after indexing is complete. Over the weekend our rhodecode server went out of diskspace because of >10GB of whoosh index files being left behind in /tmp.(rhodecode 1.7.1).

  2. valentijnscholten

    I think it finishes succesfully:

    2013-10-12 07:07:27.982 DEBUG [whoosh_indexer] >> COMMITING CHANGES TO FILE INDEX <<

    No more logging after that until the next index run.

    I see a ticket about tmp files in the whoosh tracker, but I am not sure if that's talking about temp dirs only or also temp files. In my opinion whoosh should remove the files when done.

    https://bitbucket.org/mchaput/whoosh/issue/48/temp-directories-are-not-deleted-when

    Maybe an upgrade to newest whoosh?

  3. Log in to comment