IMAP import with folders, chapter 2: folder views and permissions

[Rico 'Xaneorath' Tröger]

Hi!

I am currently tasked to work on the issue 'https://bitbucket.org/jsuto/piler/issues/841/imap-import-with-folders', which was not resolved back then, but where priorities shifted for a time, so we could not work on the problem further and keep the discussion alive.

Current Situation:

• Installed Piler Version 1.3.3

• When running 'pilerimport -i [Servername] -u [someuser] -p [thatuserspass] -R' mails become correctly imported and assigned to an imported structure of folders.

The Problem(s):

a.) Users won't see any of their mails when logging in. (Probably because they are all in folders and they don't have any folder permissions)

b.) Folder access does not appear to keep user restrictions alive (i.e. user '[e-mail]' should only see his own mails in folder 'somefolder', even though he got folder-permission for that folder.)

c.) Users should automaticaly get access to folders which got some of their mails added during import. (i.e. 'testmail' of user 'someuser' got added to the folder 'Inbox' during import -> the user 'someuser' got automatic folder-permission for 'Inbox' during import)

There also appears to be some bug related to adding folders to users in the WebGUI. If I try writing something to "Folder Search" in "Users"->"Edit", it believes the Input to be a Login-Input and offers Example-Users (not possible Folders), writing saved passwords into the password-field below, if I select a user. If I try adding Folders In the Textbox (either by Name or by ID), he states my changes to be submitted correctly on update, but there won't be any changes if I reload the "Users"->"Edit"-Page or in the database. Interestingly enough it is possible to add Folders if I log in with an User and use the "Folder" Tab there. However, it does not seem to be possible to add access to already existing folders at that point. I was able to add Folder-Access by just manualy adding 'User-Id to Folder-Id' relations in the database, but then we got the above mentioned access error b.).

I do hope we can work together to find a solution for my issues. I'm from the area of web development myself (SQL, PHP, HTML, AJAX, etc), but mostly using them plain from scratch or with CMS, right now I'm still trying to figure out the "riddle of the Sphinx", to figure out if I can fix issue b.) by changing the Sphinx-query a bit. I also have a hunch that I'll have to do some C-Programming to fix those issues a.) and c.) during the import, but maybe I'm just missing out some option. I checked on those files from per_user_folder2 you mentioned, but realised how strongly different the file model/search/search.php seems to be in build 1.3.3 compared to 1.2. And to be honest, I'd prefer if I we can find a solution for the most recent version.

Any hints, pointers and ideas where I can work are welcome, and in turn I'll of course share my findings/changes with you. For issue b.) I'm currently trying to understand Sphinx and wondering if adjusting the search-queries in model/search/search.php around line 210 will help. For issues a.) and c.) I did not find good places to start yet.

[Janos SUTO]

Before jumping in: why do you need the folders feature in the first place?

[Rico 'Xaneorath' Tröger]

We want to archive the situation of our mailclients as accuratly as possible. Right now we got a lot of (personalized) subfolders on each user-account, and we want to have this structural information in our archive as well. Also our staff should be able to access their archived mails as well, and we want to enable them to find them exactly in the same structure of folders they used.

[Janos SUTO]

Well, the folder concept in piler is somewhat different than an imap folder. Piler doesn't store imap folder information, because it stores emails in a very different structure.

The origin of the folder feature is from an unknown man for the following requirement: provide access to emails based on folders (=directories) no matter of what the email address is in the email itself. Probably some lawyer company wanted to give access to employees based NOT on the email addresses.

Later there was a request to change this behaviour to meet the requirements of another user. It was a slow process, and revised the folder behaviour of the folder feature, and it's not compatible with the old behaviour. The bottomline is that the per_user_folder2 branch is the result. Note that its source code is based on the by now outdated 1.2.x series of piler. I wanted to announce to the piler users that this is an incompatible change, give them enough time to prepare, etc. However, this merge has never happened (at least not yet).

I consider this feature sort of a side project, and I don't promote it, and I doubt that you really need it. Anyway, if you insist, then I recommend you to do a test with the per_user_folder2 branch, and check if it suits your needs. If I ever work on this feature again (though it's not on my current agenda), then it's very likely to be as it works in the per_user_folder2 branch.

[Rico 'Xaneorath' Tröger]

Well, it doesn't matter how piler stores the mails internaly, it's about the question how much information we want to be archived. Right now we store Emails and their attachments, but with a view on folders we can also store information about a hierarchical structure the mails where once in context with. I.e. a user stored all mails regarding a special sales campaign in a folder "Inbox/campaign1" on his mail account -> Why not keep that information alive when archiving? It's just an additional layer of save-able information, with the useful side-effect of keeping the work-environment familiar to the user, who might want to access some information from a long archived mail.

[Rico 'Xaneorath' Tröger]

Well, thanks for the answer anyways, it helped me to "fix" issue b.)

Apparently the unknown man put a line of code into the mail_address_filter to circumvent it and let people see any mails in a folder. So, commenting the following line out will restore usual functionality in folder context:

if(ENABLE_FOLDER_RESTRICTIONS == 1) { return ""; } [ Line 82 in model/search/search.php , Piler v 1.3.3 ]

On to the more tricky tasks a.) and c.) (Which are kind of the same task actualy.. the more I think about it)

[Rico 'Xaneorath' Tröger]

So, I managed to understand C enough to compile a working version of Piler 1.3.3 which satisfies our specifications. The new version has the following changes:

• Normal access restrictions on folders in the web-GUI (i.e. if user [e-mail] would have access to folder something.somefolder, he would still only see emails which are activated on his user profile or for one of his groups)
• Additional hierarchical level when importing folders, based on the username (i.e. a structure that would have been created as "folder1.folder2.folder3" for import-user "[e-mail]" before would now be created as "[e-mail]3". This is for seperating the Folder view for auditors (which surely wouldn't want to see one INBOX-Folder with mails of 30 different users)
• Automatical Folder-Access during import for any user who has the Import-User-Name-Mailadress in his profile or is in a group which has access to that mail-adress. (Only works if the import -u User corresponds to the mail-adress used)

I managed those changes by editing the files: * /src/config.h * /src/imap.c * /src/piler.h * /src/import.c * /model/search/search.php

Since Piler is Open-Source, I would be willing to share my work with you. If you want to take a look at the changed version, please prompt me on your prefered method of transmission for those files.