negative regex does not work in archiving rules

[Michal Mistina]

Hello Janos.

I am testing the following. According to http://www.regexr.com should string ^(?!.greenband.sk).$ match everything which does not contain greenband.sk. For references I used this site and this site. I put aforementioned regex string in the fielt From: in archiving rules. I tested it with pilertest on the following message:

[root@mailpiler regextests]# pilertest blocked_341.eml 
locale: en_US.UTF-8
build: 877
parsing...
post parsing...
message-id: <005401cf91d1$f95d7e00$ec187a00$@external.net> / 263ee49b07f75d05d7c03b650957888e3da2b2f9c6108870663c25f224738dfd
from: *Lolek [e-mail] lolek external net  (external.net)*
to: *admin [e-mail] admin mil sk  (mil.sk )*
reference: **
subject: *blocked_341*
body: *This is a multipart message in MIME format - NextPart 000 0055 01CF91E2.BC *
sent: 1403850923, delivered-date: 0
hdr len: 1354
body digest: d854416075aa625960506e71edff00e8088f30e0d645dd272361a4a6839e9c40
rules check: (null)
retention period: 1625130466
attachments:
direction: 0
spam: 0

If I put contents of the field From: to RegExr, it works and matches that string. Does mailpiler have such restrictions that not all regular expresions work in it? The test was performed on the master branch 39a2e6a306b7.

[Michal Mistina]

I guess bitbucket reparsed the regex string I've mentioned. Here is the correct one:

^(?!.*greenband\.sk).*$

[Janos SUTO]

check out the TRE regex docs: http://laurikari.net/tre/documentation/regex-syntax/

I think you should try [^somethinghere] structure.

However be aware that such a rule would discard all emails that don't come from the *greenband.sk domain (provided that the regex is correct).

[Michal Mistina]

It's OK for me to block every e-mail which comes from other domains except greenband.sk. That is the target. I will narrow this rule by defining To: field. This is one little step towards the requirements :-)

So I've tried more things.

1. I am testing with the same e-mail message as before.

2. Set archiving rule - From: field to [e-mail] --- matched

3. Set archiving rule - From: field to [^[e-mail]] --- matched

[root@mailpiler regextests]# pilertest blocked_341.eml 
locale: en_US.UTF-8
build: 877
parsing...
post parsing...
message-id: <005401cf91d1$f95d7e00$ec187a00$@external.net> / 263ee49b07f75d05d7c03b650957888e3da2b2f9c6108870663c25f224738dfd
from: *Lolek [e-mail] lolek external net  (external.net)*
to: *admin [e-mail] admin mil sk  (mil.sk )*
reference: **
subject: *blocked_341*
body: *This is a multipart message in MIME format - NextPart 000 0055 01CF91E2.BC *
sent: 1403850923, delivered-date: 0
hdr len: 1354
body digest: d854416075aa625960506e71edff00e8088f30e0d645dd272361a4a6839e9c40
rules check: domain=,from=[^[e-mail]],to=,subject=,size>0,att.name=,att.type=,att.size>0,spam=-1
retention period: 1625141879
attachments:
direction: 0
spam: 0

Why in the point 3 the rule matched? The laurikari web page says "If the list begins with ^ the meaning is negated; any character matching no item in the list is matched."

I don't understand then. I thought that 3rd point should match only if there is not "[e-mail]" inside From: field in the e-mail message. I expected (null) match.

I also tried [^[[e-mail]]] without success.

[Michal Mistina]

I'm leaving for few days from today, so I won't be here to test it. Next week, so please wait till you close the issue due to inactivity. Thanx.

[Janos SUTO]

OK, I won't close it, and I will do dome tests with the "negative" rules.

[Michal Mistina]

Janos, I am back. Did you try something with "negative" rules?

[Janos SUTO]

Yes, but not much success.

[Janos SUTO]

Today I experimented with it, but still no luck, so I contacted the tre developer for help. Hopefully he will help us to fix the regex string.

[Michal Mistina]

Thank you for your struggle. I am very interested in the response.

[Michal Mistina]

I tried to experiment with regex matching in subject field. E.g.

"subject: *test351*"

matches regex [test351] and everything else e.g.

"subject: *x test351*"

matches [^test351]. That means negative regex is working. The issue is, that if I want to match all other incoming domains except one (e.g. greenband.sk) I have to compare it to the whole string which is inside the From: field. I will take an example of From field from pilertest:

from: *Lolek [e-mail] lolek greenband sk  (greenband.sk)*

The steps to achieving this could be these:

• Match everything that matches following string:

Lolek [e-mail] lolek greenband sk  (greenband.sk)

• Negate it.

I can match everything with this not very clever regex:

.+?\ .+?@greenband\.sk\ .+?\ greenband\ sk\ +\(greenband\.sk\)

But I don't know how to negate it all, so if it is for example:

Lolek [e-mail] lolek external net  (external.net)

it is hit whole by pilertest.

Easier way to negate everything would be to implement it in the code of mailpiler. For example to create check box with the name "not" next to the form boxes of archiving rules. In that case the whole regex string in the form box would be negated if you check "not" next to it.

[Janos SUTO]

Indeed, it's definitely a way to do so, however you may setup several parameters (from, to, subject, size, attachment, etc.) for a single rule. The if all set conditions are met, then it's a match.

But let's say you create a new rule with a from and a size value. Then how to evaluate it when using a NOT (or inverted) logic?

Anyway it would be far more manageable if it could be fixed with the tre regex stuff. Unfortunately no response yet from the developer.

[Michal Mistina]

Can you please post an example?

[Janos SUTO]

Eg. rule #1: from: @domain.com, size: >20000

The point is: how to evaulate the negated expression if there are two (or more) conditions?

Eg.

message #1: from=[e-mail], size=23000 message #2: from=[e-mail], size=15000 message #3: from=[e-mail], size=23000 message #4: from=[e-mail], size=15000

So which message should be discarded by the negated rule #1?

[Michal Mistina]

I think only the message no.4 will be archived and other messages will be discarded. I don't think size should have "NOT" checkbox available. How is the result of the rule built? I thought there is a logical AND between individual conditions.... !(From) AND (Size). Or you meant to negate the whole rule after it is built? .... !((From) AND (Size))

Currently how the rules are handled? Like in firewalls? The first match ends of the evaluation of the rules that are behind?

Sorry for confusion. I am not a developer, so I don't see behind the scenes.

[Janos SUTO]

Still no response from the tre developer, and I'm a bit reluctant of complicating the rule processing with a negating logic.

Anyway when a message is processed all the rules are checked until there's a hit (just as with firewalls like you mentioned). There's no specific order, just as the sql server presents the rules.

[Michal Mistina]

All right. Do you want to leave this issue opened till the tre developer replies? From my perspective there is no other way to solve it. Only political way - to say customer, that he needs to archive both outbound and also inbound e-mails. No problem in that. Maybe if somebody other asks for it, you can now save it as a feature request ;-)

[Janos SUTO]

I'm not sure if he will reply at all. Even I reply within a few days :-) However if you described your environment then I may help you to achieve it, perhaps without complex rules. For example (eg. with postfix) you may choose to archive only incoming emails only.

[Michal Mistina]

Thank you for the offer, but the issue is, that there is a black box where we cannot influence the behaviour of it. It just journals everything - sends everything somewhere (mailpiler). The vendor of the black box is going to add such filtering on outgoing/ingoing mails. But this can be released in years, because it's only a feature request. I tried to solve it on the mailpiler. According to these difficulties we can close this case and if the tre developer reponds it can be reopened. Thank you again.