- edited description
Amazon S3 and Mailpiler
Issue #582
resolved
Hello,
I have a 2 questions to see if either are possible. Currently I have my Amazon S3 drive mounted to my CenOS server. Mailpiler is running well except it is sucking up my local storage. My questions are the following:
Is it possible to relay the emails to my S3 and have sphinx index them? or Is it possible to move my database and sphinx onto the S3?
I am open to other suggestions of course. My main goal here is I want to use S3 as my primary storage and still have the GUI looking nicely with fresh indexes.
Comments (17)
-
reporter
-
repo owner
If you can mount the S3 drive to be seen as a normal partition, then it should work. I'd put it under /var/piler. Not sure about the latency you might experience. Mysql on an S3 drive might be tricky, check out the mysql docs about it. Another option can be emails on S3, mysql db on local storage.
Anyway let me know how it works.
-
Keep the MySQL on local host, MySQL never works correctly when it's mounted on remote storage.
As for mail storage, it will lag immensely if it's on s3, and don't start the piler services if the storage is not mounted :P
Use one of the following, pretty much test which works best for your OS and network.
https://github.com/skoobe/riofs
-
repo owner
- changed status to resolved
Thanks, extremeshok :-)
-
reporter
Well, how exactly would I have the emails be sent to s3? I have s3 mounted but will have to move the directory to /var/piler. If the emails are on my s3 will sphinx index them into the gui?
-
repo owner
You don't "send". Rather mount the s3 storage to your host, thus it acts as a partition on your computer. Sphinx indexing has nothing to do with how / where you store emails. Sphinx reads the data to be indexed from a mysql table.
-
reporter
If I mount s3 to /var/piler how will I get it to show up in the GUI? Will the emails just spill over to the partition or will it save local until the storage is full then move to the partition?
-
reporter
If I mount s3 to /var/piler how will I get it to show up in the GUI? Will the emails just spill over to the partition or will it save local until the storage is full then move to the partition?
-
The s3 will be running on FUSE, basically it is a partition mounted as /var/piler
There is no spill over.. As far as everything is concerned its a physical hard-drive connected to the system.
-
/var/pilerwould be a mount point on Fuse S3.. Therefore all emails would be on S3.
Ie. 100% S3 based storage.
There is no "simple" way to run dual storage archives, ie. 90% on S3 and 10% on /dev/sda1.
-
reporter
I understand, currently my s3 is not mounted in /var/piler so I need to move it to that. I had followed this guide to do it and I believe it is mounted in its own directory.
-
reporter
This was what I used to set it up http://tecadmin.net/mount-s3-bucket-centosrhel-ubuntu-using-s3fs/
-
Stop piler Rename existing /var/piler to /var/piler-old Mount s3 to /var/piler (you could also create a link) Copy the contents of /var/piler-old to /var/piler
Start piler and the mails will be served via s3.
You will need to enforce piler does not start if s3 is not mounted.
Simply Move the files from the old dir to the new one, make sure your move the permissions and ownership as well.
Then
-
reporter
Currently, s3 is mounted to /var/piler. When I hit df it shows my s3 is moiunted on /var/piler/s3mnt which is correct. I modified the permissions of the s3 mount to allow piler to read/write and I changed the owner and group to piler. Nothing is being written to s3 though and when I load up the gui it does not even show up as a storage option. Any ideas?
-
reporter
If I go back to the /var/piler it seems s3mnt permissions reverted to root root upon mounting the drive.
-
Mount s3 as /var/piler/ not /var/piler/s3mnt
-
- Log in to comment
