multiple storage

[Crzaccount]

Hi Janos,

  I would like to split the piler data into multiple storages / volumes instead of single storage.

Eg: I am having 500 GB HDD. 500 GB HDD am using for 2 years data. I want to move 2 years data to USB HDD & keep only the current 2 years data in 500GB HDD.

Whenever i need the data, i want to retrieve it from USB disk.

Is there any possibility for the above ?

[eXtremeSHOK]

This can already be done on the file system directly.

Move the data files older than 2 years to a different mount point and use symbolic links.

Files older than 2 years won't load as they can't be read off the file system.

When u mount the USB, they will be able to be read

[Janos SUTO]

Currently no. I think it may be done, however it poses some challenges (besides to redesign the way piler stores stuff).

I know that some other products use 'volumes' to store messages. Such a 'volume' consists of a fixed number of zip files, and the products put the stored messages to zip files (1 message to 1 zip file). I admit these files are easier to move to a secondary media, etc, though I'm not sure if there's some performance penalty since the use a zip files (or other structures). Another challange is how to tell pilerget that the location of a message is changed from the HDD to the USB disk? Perhaps to store the volume serial in the metadata table, and define somewhere the path of the given volume.

Another concept could be sparse files. You can create a single file acting as a volume, and by careful positioning you may put messages inside the sparse file. Thus you have a single file which is easier to handle then zillions of small files, though you have to make sure that your backup software (if you use any) can handle sparse files effectively.

One thing is sure: whatever we pick, it will pose an extra layer adding (some) complexity, though I can see the advantages of another method as well. Not to mention that any change requires lots of testing to deem it stable. Anyway I'll bring this topic to the mailing list, let others share their thoughts, feel free to join.

[Former user]

I believe that this could be done with the same concept that Oracle uses, tablespaces.

We could create a "mailspace" and link it to piler, piler will store messages in these mailspaces. As all metadata is stored in the database, whenever a message needs to be loaded it could check if the mailspace is available or not.

Adding another layer, like ZIP, sparse file or like, is a waste of resources in my opinion.

[Janos SUTO]

Hmm, currently piler stores emails to /var/piler/store/00. This could be a link to /opt/mailspace0, and you may create another one, eg. /opt/mailspace1 to /var/piler/store/01, etc. 00, 01, ... are controlled by 'server_id' variable. Perhaps it should be renamed to mailspace or similar. (Provided that everyone left it as the default: 00).

[eXtremeSHOK]

One could do the following. Add 3 columns to the mail table, path, binary_path

This would be the simplest way to allow for a storage change as the path and binary_path would use the existing piler locations and update the records.

Assume the following: mails stored on ssd: /var/pilermail/ binaries(attachments) stored on hdd: /datastore/

One could then move/export the mails and/or binaries for X domain or all domains for all or a specific month or a specific year to a different location.

Example: pilermove --domain="extremeshok.com" --month=all --year=2013 --type=binary /coldstorage/extremeshok/2013

This would move all attachments for extremeshok for the year 2013 to the new location (/coldstorage/...) The database table for the mail would update the binary_path

pilermove --domain=all --month=all --year=2014 --type=all /coldstorage/2014

This would move all mails and attachments for year 2014 to the new location (/coldstorage/2014) The database table for the mail would update the path and binary_path

Then the mails are loaded/viewed/exported/accessed in piler, they would use the path to and the binary_path to view/fetch the emails.

If one wanted to use a similar logic to a maildir structure.

pilermove --domain=all --month=all --year=all --type=mail--format=maildir /var/pilermail/

Mails would then be moved to the following structure. /var/pilermail/domain/year/month/

pilermove --domain=all --month=all --year=all --type=binary --format=maildir /datastore

binaries(attachments) would now be stored like this /datastore/domain/year/month

[Janos SUTO]

Your idea is pretty impressive, though keeping emails and attachments together would simplify the job. Though adding yet another varchar(255) default null column definitely increases the size of the stored sql data. So I'd rather use a number to specify the 'volume' (0-255), and you can keep the volume info in another table (volume). The drawback is that you need an extra select for storing each emails.

[eXtremeSHOK]

Yarr.. thats what im here for ;-P

Adding a volume table is better. Also paths can be longer than 255, so it would make more sense.

Assigning an storage path ID (int) is the way togo.

As always i would prefer mails and binaries split. SSD;s are better for small files. HDD"s are beter for large files.

Regarding the extra select, one could do a join and use 1 query (can write for you if you need).

However it might be faster to do a single slect and return an array.

Then in the code just cross reference the array to get the path.

$mail_path[$email->storage_email_id]

$mail_path[$email->storage_binary_id]

[Crzaccount]

Hi Janos,

As suggested i had moved the data to /opt/mailspace & created the link in piler/store/00 . Changed the server_id in piler.conf.

If I add another volume (mailspace1 --> 01) with the permission piler:piler, new mails are storing in 00 only. I want to store the mails in 01. Any possibility ?

[Janos SUTO]

Then try setting server_id=1 for now.

[Crzaccount]

If i change the server_id=1, am not able to view the mails which is in 00. ie. Am getting x mark (Message Failed Verifitcation) while search.

[Janos SUTO]

You are right, sorry, change it back to 0. Then unfortunately you can't achieve your goal at the moment. However you can do something close to your goal. You can see directories under /var/piler/store/00. Move all of them but the last one to /opt/mailspace0, then create a symlink to them under /var/piler/store/00.

[Janos SUTO]

I hope the suggested solution is close enough to acceptable.