Some Improvements to Docker+Documentation

[Alexander Kaplan]

Hi Janos,

first, thank you a lot for all the work you put here in, mailpiler is amazing, im using it here for myself for my own mailserver/domain, for a long time already.

And i switched recently to the docker based builds, which are working really good either, except the :jammy build, which needs some deb package in the /data folder.

‌

However, some improvements could make lives easier:

1. (Docker) Updating is pretty confusing, since the /etc/piler folder is consistent mounted, means that changes in /etc/piler should be documented, in a changelog or somewhere, like this ones:
   https://bitbucket.org/jsuto/piler/commits/197f09ed47e4f3cebcb0ecf031f2f58da4b3ecc3
   https://bitbucket.org/jsuto/piler/commits/d4a57e386c05300bed6ed89ff6e78e9e3990b95b
   (thats just examples)
2. (Docker) could you change the location of /var/piller/www folder to somewhere else? because thats the only folder in /var/piler that can live in the docker image itself, to get deleted/recreated when you update docker? That folder has no configurations that needs to be changed at all.
   Alternatively we can mount all other folders around that one, which would make a lot mountpoints, however, i fear that this way the www folder will simply get deleted on updating and wont get recreated with updated files.
   There is only /var/piler/www/tmp, where im not sure if this needs to be persistent across updates.
3. (Documentation) The Documentation lacks a bit, for myself everything is clear with trial and error approach.
   But i would make easier examples in the Documentation about retention Rules/groups/, like some pictures from the gui and describing what the rule does. Mantioning that “if a field is left empty, it doesnt get matched” and in From/To @mydomain.com will match every email to or from this domain, it has not to be a specific email address…
4. (Docker) Is there an option possible to add an Watching directory? since i have an nfs share mounted directly to /var/piler/store and the GUI shows only / . This isn’t important, since its only a visual aspect, but would be nice to have.

With the Documentation Part, i can help there if you want, to provide at least some sort of commitment. Since im a pretty worthless user that uses only mailpiler, without helping etc… :-(

About the new UI, if its the one on your Demo-Page…
Im sorry, but its pretty ugly compared to what we have already :-(
The Admin Page is definitively better on the Old GUI, but the Auditor/User Page is actually better on the new GUI.
I like the new Layout on the User/Auditor Account, its just not very visually appreciating somehow xD

‌

Cheers and Thank you a lot!

[Alexander Kaplan]

I found a small issue either:

After applying this patch:
https://bitbucket.org/jsuto/piler/commits/197f09ed47e4f3cebcb0ecf031f2f58da4b3ecc3

/usr/libexec/piler/indexer.main.sh
FATAL: table 'dailydelta1': cannot apply kill list to myself: killlist_target=main1:kl, main2:kl, main3:kl, main4:kl, dailydelta1:kl

‌

Made probably by myself since i shouldnt apply, but anyway, maybe that helps.

[Janos SUTO]

Hello Alexander, thank you for your feedback. It’s ok not to like the new gui, I’ll try to improve the admin page. Anyway, I’ll address some of your ideas now.

Docker #1: there’s a RELEASE_NOTES file in the source code root dir that mentions the 2nd manticore config change. You are correct that the first commit is not in the release notes. I’ll try to do a better job in this regard in the future.

Docker #2: the Dockerfile lists in the docker dir lists 3 volumes you need. In fact all dirs in /var/piler can be in the container except store and manticore subdirs. NB: be sure to check /var/piler/error dir for any stale emails before upgrades. So I believe that the current volume scheme is OK.

Docker #4: Add these config options to /etc/piler/config-site.php:

$partitions_to_monitor = array('/', ‘/home', '/var', '/var/piler', '/var/piler/store’, '/tmp');
$config['DATA_PARTITION'] = '/var/piler/store';

Documentation #3: I’ve added your comments to the docs

[Alexander Kaplan]

Cool, Thanks for the answer :-)
You are absolutely correct, actually i need only manticore and store. The only Problem im not 100% sure, is if the other dirs get recreated on update. (I do an image remove and docker run) on update.

However, i will try that today out with an new testing instance of piler.

‌

The only question that is left, is about the killlist_target, as far as i understood i need it to delete emails for pilerpurge or purge.sh (cron).
But that means also, that if i add the killlist_target variable to the manticore.conf, i have to (recreate manticore index?)
1. disable cron
2. empty out manticore directory
3. su -c "indexer --all --config /etc/piler/manticore.conf” piler
4. su -c "reindex -c /etc/piler/piler.conf -a” piler
5. su -c "/usr/libexec/piler/indexer.delta.sh” piler
6. su -c "/usr/libexec/piler/indexer.main.sh” piler
7. enable cron again.

I did recreated the indexes already earlier this way, if i had some sorts of issues and transferred with pilerexport/pilerimport already my mails to new installations, switching from native to docker based instance for example…
worked actually always very flawless, its just an very long task xD
Im not sure about the mariadb database, if the database can be cleaned out either and if reindexing adds the entries back to the mysql database. But as far as i understand, the database is only needed manticore/sphinx.
Ultimatively, i would like to create an “rescue script/cleanup” script, that empties the mysqldb and manticore and recreates it, because i feel somehow that i have a lot of crap in the mysql database.

‌

However, thanks Janos for all the work, you don’t have to answer my questions, i can find those things out myself by trial and error either :-)

Cheers

[Janos SUTO]

I think you don’t have to recreate the manticore index if you add the killlist stuff. Note that the mysql piler database is crucial, you can’t get rid of it as it contains important metadata. Without it your archive won’t function. The reindex tool is only meant to recreate or heal the manticore index data. Currently there’s no such tool that re-populates even the mysql tables using the stored archived files.

[Alexander Kaplan]

Hi Janos, thanks for the answers! :-)

I try to figure out then what it needs to supress the killlist error:
/usr/libexec/piler/indexer.main.sh
FATAL: table 'dailydelta1': cannot apply kill list to myself: killlist_target=main1:kl, main2:kl, main3:kl, main4:kl, dailydelta1:kl

When i figured it out, i will write then a solution here.
EDIT: But it will take 2-3 days, since i have a ton of other stuff to do (for the work), the next 2 days

Cheers

[Janos SUTO]

No worries, until then the issue is closed.