Help regarding piler storage

[Gagan Bansal]

Hi,

We’ve got piler setup in my organisation and we are facing an issue wherein my storage drive (which is a magnetic ebs on AWS) is running out of space. I want to move the piler data onto another EFS storage that I’ve created. Can someone help me with what are the folders that I need to move and what should be the proper procedure? So far, I’ve been thinking of mounting both onto my EC2 instance and using rsync. Will this be enough?

Please note that once I’ve moved the existing data I’ll be changing the default storage location to the EFS. Also, the emails are all upper mgmt and client related and hence we need to have them backed up as we need the data for regular audits.

[Janos SUTO]

Hello, piler stores data at 3 places:

• /var/piler/store: emails
• /var/piler/sphinx: index files
• /var/lib/mysql: the mysql database

Most of the time store and sphinx dir size is significantly larger than mysql, so you may (or may not) keep the mysql database as it is, and copy only the other 2 directories. Btw. you should also consider the current partitioning scheme, eg. is /var or /var/piler a separate partition, etc.

[Gagan Bansal]

Thanks for the details. Yes, I do have /var/piler as a separate partition on my EC2 instance. I have mounted my EFS storage as another partition (/var/efs) onto my instance. I am trying to rsync the data from my /var/piler/ into my /var/efs mount. Once that is done I will change the mount point for my EFS to /var/piler. Will this cause any issues? Also, what actions should I take before doing the rsync operation? Should I stop the piler and sphinx search services?

[Gagan Bansal]

Also, this is the current directory structure in /var/piler:

drwx------. 7 piler piler 4.0K Jul 20 2016 imap
drwx------ 2 root root 16K Jul 8 2015 lost+found
drwxr-xr-x. 5 mysql mysql 4.0K Feb 20 00:24 mysql
drwxr-xr-x. 2 piler piler 4.0K Jun 5 16:15 sphinx
drwxr-xr-x. 2 piler piler 4.0K Jul 9 2015 stat
drwx------. 3 piler piler 4.0K Jul 7 2015 store
drwx--x--x. 2 piler piler 1.5M Jun 5 16:15 tmp
drwxr-xr-x 14 root root 4.0K Feb 20 00:15 www
drwxr-xr-x. 14 root root 4.0K Apr 5 2017 www.old

Is it possible for someone to analyse and tell me what the stat, tmp, www and www.old folders are for? Also, the mysql folder present here is almost 26G+ in size.

Any help would be appreciated.

[Janos SUTO]

You are funny. Someone renamed www to www.old and you ask someone to “analyse” it?

www dir should hold the piler GUI. www.old might be a previous revision I guess. And it seems that /var/lib/mysql is moved or symlinked to /var/piler.

Before you ask: you need all these stuff on the new data partition as well. Be sure to stop piler, searchd and mysql daemons before starting to copy.

[Gagan Bansal]

Sorry, I wasn’t thinking when I typed that statement. Thanks for clearing that up. Yes, the mysql library in /var/lib/mysql has a link to /var/piler/mysql. I have stopped the daemons as per your instruction and have started copying. If I come across any issues I will update accordingly.

[Janos SUTO]

Sure, in that case be sure to reopen this issue.

[Gagan Bansal]

I’ve finished the movement of the files to my EFS. To be safe I ran a full reindex of the archive using reindex -a which took almost a full day(there were about 26218273 mails). Also, I disabled the cron jobs as mentioned in your FAQ. Now when I search for emails in the WebUI, it shows me the count but the actual display for emails remains empty. I’ve had to revert to the magnetic storage which was working perfectly. Can you please let me know what can be done to resolve this issue? I’ve inherited this piler setup from the previous guy and have next to zero idea about piler troubleshooting so any and all help would be appreciated.

[Janos SUTO]

Check out the mail log to see what’s going on, search for pilerget, piler should syslog any issues.

‌

Alternatively if you want me to check out the archive, then visit https://mailpiler.com/support/ and order commercial support.

[Gagan Bansal]

Hi,

The issue is that we are unable to see the email content in the display window even though the counter shows 1-20 of 700 or something like that. The search is unable to find even the previously archived emails. Could it have something to do with the php files? Should I try rebuilding the index again to check?

[Janos SUTO]

I understood the issue the first time. If the upper pane shows you messages, then the sphinx index data is fine.

If you can’t see the email content in the lower pane, then it’s a pilerget utility issue. You can either check the mail log, and figure out and fix for yourself what’s wrong, or you may use the support service shown above.

[Janos SUTO]

No news is good news.