Cannot get pilertest work

[Michal Mistina]

Hello Janos.

I've downloaded latest master branch (jsuto-piler-a6ff3bf0985a) and installed it over the piler-0.1.24. Of course I did "make uninstall" on the old piler version. I am trying to match just the simple condition. In the field "To", there engine should match messages sent to domain "mil.sk", so they will be dropped. It is just the test while the request is for more difficult regex.

So I've created the archiving rule where field To is filled with string "mil.sk".

I've restarted piler service.

The message contains this in the field To:

To: "'user'" [e-mail]

The result of "pilertest message.eml" is:

locale: en_US.UTF-8
build: 875
parsing...
post parsing...
message-id: null / 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b
from: * ()*
to: * ()*
reference: **
subject: **
body: **
sent: 0, delivered-date: 0
hdr len: 1394
body digest: 2880f97c256e07f7b7dba321f0b98454409a1e061895269798e9b156a6a085e3
rules check: (null)
retention period: 1624714116
attachments:
direction: 0
spam: 0
NOT IN mydomains

I guess if the pilertest matches the rule, the "rules check" field of the output is different.

What am I doing wrong?

Michal

[Janos SUTO]

Hello Michal, I assume you have updated the sql schema as well. I think either

a) piler user can't read message.eml b) piler user can't create temp files in the current directory

[Michal Mistina]

Hello Janos.

chown piler:piler on the .eml messages solved the issue where I couldn't see correct output from the pilertest. But I couldn't get the message matched the rule where To: field has the string "mil.sk" inside:

[root@mailpiler regextests]# pilertest blocked_341.eml 
locale: en_US.UTF-8
build: 875
parsing...
post parsing...
message-id: <005401cf91d1$f95d7e00$ec187a00$@external.net> / 263ee49b07f75d05d7c03b650957888e3da2b2f9c6108870663c25f224738dfd
from: *Lolek [e-mail] lolek external net  (external.net)*
to: *admin [e-mail] admin mil sk  (mil.sk )*
reference: **
subject: *blocked_341*
body: *This is a multipart message in MIME format - NextPart 000 0055 01CF91E2.BC *
sent: 1403850923, delivered-date: 0
hdr len: 1389
body digest: eb43e9e5d89450dbb01b6c3b8ce57fac04854ac38a2eb364a6ebe3310bd47eaf
rules check: (null)
retention period: 1624775843
attachments:
direction: 0
spam: 0

I remember I've applied the sql schema when I tested previous master branch release. Is there a way to check wether the sql schema is there?

[Janos SUTO]

Show me "grep To: messages.eml" output. Btw. there's no automatic method to check the sql schema other than you, in other words you have to check manually.

[Michal Mistina]

[root@mailpiler regextests]# grep To: blocked_341.eml 
To: "'admin'" <[e-mail]>

[Janos SUTO]

To: "'user'" [e-mail]

is not a match for

To: "'admin'" [e-mail]

How about putting simply "[e-mail]" to the rule? Note that just for pilertest no need to restart the piler daemon.

[Michal Mistina]

I would like to match whole domain mil.sk.

I added [e-mail] to the rule.

[root@mailpiler regextests]# pilertest blocked_341.eml 
locale: en_US.UTF-8
build: 875
parsing...
post parsing...
message-id: <005401cf91d1$f95d7e00$ec187a00$@external.net> / 263ee49b07f75d05d7c03b650957888e3da2b2f9c6108870663c25f224738dfd
from: *Lolek [e-mail] lolek external net  (external.net)*
to: *admin [e-mail] admin mil sk  (mil.sk )*
reference: **
subject: *blocked_341*
body: *This is a multipart message in MIME format - NextPart 000 0055 01CF91E2.BC *
sent: 1403850923, delivered-date: 0
hdr len: 1354
body digest: d854416075aa625960506e71edff00e8088f30e0d645dd272361a4a6839e9c40
rules check: (null)
retention period: 1624785897
attachments:
direction: 0
spam: 0

[Janos SUTO]

Please check that the current archive table schema matches the one you can find in util/db-mysql.sql

[Michal Mistina]

You mean archiving_rule table schema? I don't have in the piler database archive table schema:

mysql> show tables;           
+-------------------+
| Tables_in_piler   |
+-------------------+
| archiving_rule    |
| attachment        |
| audit             |
| autosearch        |
| counter           |
| counter_stats     |
| customer_settings |
| domain            |
| domain_user       |
| email             |
| email_groups      |
| folder            |
| folder_extra      |
| folder_message    |
| folder_user       |
| google            |
| google_imap       |
| group             |
| group_email       |
| group_user        |
| import            |
| ldap              |
| metadata          |
| note              |
| online            |
| option            |
| rcpt              |
| remote            |
| retention_rule    |
| search            |
| sph_counter       |
| sph_index         |
| tag               |
| user              |
| user_settings     |
| v_attachment      |
| v_messages        |
+-------------------+

[Janos SUTO]

Ehh, yes, you are right: I meant the archiving_rule table

[Michal Mistina]

I compared those schemas, but it seems they are identical.

..in database:

CREATE TABLE `archiving_rule` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `domain` varchar(128) DEFAULT NULL,
  `from` varchar(128) DEFAULT NULL,
  `to` varchar(128) DEFAULT NULL,
  `subject` varchar(128) DEFAULT NULL,
  `_size` char(2) DEFAULT NULL,
  `size` int(11) DEFAULT '0',
  `attachment_name` varchar(128) DEFAULT NULL,
  `attachment_type` varchar(128) DEFAULT NULL,
  `_attachment_size` char(2) DEFAULT NULL,
  `attachment_size` int(11) DEFAULT '0',
  `spam` tinyint(1) DEFAULT '-1',
  `days` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `from` (`from`,`to`,`subject`,`_size`,`size`,`attachment_name`,`attachment_type`,`_attachment_size`,`attachment_size`,`spam`)
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8

... in master branch util/db-mysql.sql:

create table `archiving_rule` (
   `id` bigint unsigned not null auto_increment,
   `domain` varchar(128) default null,
   `from` varchar(128) default null,
   `to` varchar(128) default null,
   `subject` varchar(128) default null,
   `_size` char(2) default null,
   `size` int default 0,
   `attachment_name` varchar(128) default null,
   `attachment_type` varchar(128) default null,
   `_attachment_size` char(2) default null,
   `attachment_size` int default 0,
   `spam` tinyint(1) default -1,
   `days` int default 0,
   primary key (`id`),
   unique(`from`,`to`,`subject`,`_size`,`size`,`attachment_name`,`attachment_type`,`_attachment_size`,`attachment_size`,`spam`) 
) ENGINE=InnoDB;

[Michal Mistina]

Could you reproduce the issue in your test environment?

[Janos SUTO]

Let me fabricate such a message, and I'll let you know soon

[Janos SUTO]

I think, it's a bug. I'm working on the fix.

[Janos SUTO]

OK, download the latest master branch, and try it. I think it should work this time.

Note the "to" line from pilertest output, eg:

to: admin [e-mail] admin mil sk (mil.sk )

The parser matches this string against the To field in both the archiving and retention rules. So perhaps matching for the user name in the To field is a bit trickier than it seems at first.

[Michal Mistina]

You've repaired it :-)

[root@mailpiler regextests]# pilertest blocked_341.eml 
locale: en_US.UTF-8
build: 877
parsing...
post parsing...
message-id: <005401cf91d1$f95d7e00$ec187a00$@external.net> / 263ee49b07f75d05d7c03b650957888e3da2b2f9c6108870663c25f224738dfd
from: *Lolek [e-mail] lolek external net  (external.net)*
to: *admin [e-mail] admin mil sk  (mil.sk )*
reference: **
subject: *blocked_341*
body: *This is a multipart message in MIME format - NextPart 000 0055 01CF91E2.BC *
sent: 1403850923, delivered-date: 0
hdr len: 1354
body digest: d854416075aa625960506e71edff00e8088f30e0d645dd272361a4a6839e9c40
rules check: domain=,from=,to=[e-mail],subject=,size>0,att.name=,att.type=,att.size>0,spam=-1
retention period: 1625064115
attachments:
direction: 0
spam: 0

Thank you, Janos.

[Janos SUTO]

You are welcome.