Importing mail stops processing and I have to start over.....

[My Account]

Currently when I import converted emails into piler, most of the time it works, but lately I have been halted where it gets stuck or stops at a certain number and I cannot tell why it gets locked or stops. Is there anyway to tell why it's getting held up like that? When this happens it usually stops around the same number...not always, but most of the time. When I go to kick it off again, it will stop again. I need to know how to determine the issue to fix it, so I don't keep wasting my time trying to re-import only to have it stop again.

The code I have in my bash script is the following which contains the -r to remove processed items.

clear; echo "Starting Import Process into Piler" pilerimport -d /var/tmp/import_PST/ -r done

I really need some help here as this is taking way too long to import what I need to and I need a faster method or a way to find out what is causing the failed processing.

Please advise. Thanks in advance!

[Janos SUTO]

A user reported a similar symptom, and he claimed that he had to increase the number of available open file descriptors. Though this issue is still unconfirmed.

I'd try the following. Group emails inside /var/tmp/import_PST/ to subdirectories, eg. put 10k messages to /var/tmp/import_PST/1, another 10k to /var/tmp/import_PST/2, etc. Then run pilerimport on /var/tmp/import_PST/1, and check if it actually removes the imported messages properly. Then move on to /var/tmp/import_PST/2, etc.

Let me know how it works. Btw. what piler version do you use?

[eXtremeSHOK]

It's basically file system and Linux processing limits

We have written a commercial script which will get around these issues as well as speeding up the import approx 4x.

[My Account]

No....so I'm not accepting this answer as I've been doing the same thing for over a year now and just recently it's not working. So there is something else wrong and I'd like to view some kind of log to tell me why it stopped processing imports. Is there even a log?

Also, what has changed is that you have had me use the -r to delete anything that is processed. It appears that it will not delete any duplicates it finds so really it is not deleting everything it touches.....is there a way to delete even the duplicates during the processing?

Thanks,

Mike Lee

[eXtremeSHOK]

Enable the debug switch and pipe the output to a log.

Also check your respective kernel, sphinx and mail logs

[Janos SUTO]

You are right, it seems that when processing a local directory, duplicates are not removed. To fix that edit src/pilerimport.c, go to line 235 which reads as

if(data->import->remove_after_import == 1 && ret != ERR) unlink(fname);

and change it to this:

#!c

if(data->import->remove_after_import == 1 && rc != ERR) unlink(fname); ```

Then recompile, and run the import again.

[Janos SUTO]

No news is good news.

[My Account]

No news is not good news. I've been out for a while. I'm trying to get someone with more Linux experience to help me figure out how to edit and compile what you asked me to, because I myself am not a Linux guy and your instructions are less than verbose.

I also want to know if there are known issues with this sort of thing with the current version that was installed. Version is Piler 1.1.0 Build 884, are there updates or upgrades to be made with this version that may fix my issue?

And I cannot seem to locate what the "debug" switch is in any of the documentation I am looking at on your site. Please advise.

[Janos SUTO]

1.1.0 is not the current version. It's 1.1.1 or you may even try the master branch, ie. the latest developer version. If you can't get somebody to do the fix for you, then I may help you via teamviewer. Contact me at my email address if it's ok for you, see piler -V.