differences between log data and meta data regarding "from:"

[Uwe Kiewel]

What kind of "from" is logged by piler to syslog and what kind of "from" is stored in database?

maillog entry from piler:

Jun 14 22:03:54 duckburg piler[28909]: 40000000594196b419a5b644006c5a11bf47: from=[e-mail], size=26630/8821, attachments=0, reference=<[e-mail]>, message-id=<[e-mail]>, retention=366, folder=0, delay=0.05, delays=0.01/0.00/0.03/0.00/0.00/0.00, status=stored

Metadata from v_messages:

mysql> select * from v_messages where piler_id = '40000000594196b419a5b644006c5a11bf47'\G
*************************** 1. row ***************************
        id: 990
  piler_id: 40000000594196b419a5b644006c5a11bf47
      from: [e-mail]
fromdomain: schaufler-ca.com
        to: [e-mail]
  todomain: redhat.com
   subject: Re: [PATCH 08/27] VFS: Introduce the structs and doc for a filesystem context [ver #5]
      size: 26630
 direction: 0
      sent: 1497470623
  retained: 1529093034
   arrived: 1497470634
    digest: 5cd1408f9f7307995385e68bffc5aa1dfca4f9a04f412c109caeaaad915f442c
bodydigest: 283c0d3aa4d851f87cf3b31fb1791eae32df7b9ab6e5502cf34fa349dcfb0a36
   deleted: 0

header of this mail:

Received: from localhost (localhost.localdomain [127.0.0.1])
        by duckburg.kiewel-online.name (Postfix) with ESMTP id 46A6D2A18EA
        for <[e-mail]>; Wed, 14 Jun 2017 22:03:54 +0200 (CEST)
Received: from duckburg.kiewel-online.name ([127.0.0.1])
 by localhost (duckburg.kiewel-online.name [127.0.0.1]) (maiad, port 10024)
 with ESMTP id 27711-07 for <[e-mail]>;
 Wed, 14 Jun 2017 22:03:50 +0200 (CEST)
Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
        by duckburg.kiewel-online.name (Postfix) with ESMTP
        for <[e-mail]>; Wed, 14 Jun 2017 22:03:49 +0200 (CEST)
Received: ([e-mail]) by vger.kernel.org via listexpand
        id S1752040AbdFNUDq (ORCPT <rfc822;[e-mail]>);
        Wed, 14 Jun 2017 16:03:46 -0400
Received: from nm10-vm0.bullet.mail.ne1.yahoo.com ([98.138.91.72]:53518 "EHLO
        nm10-vm0.bullet.mail.ne1.yahoo.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1751760AbdFNUDp (ORCPT
        <rfc822;[e-mail]>);
        Wed, 14 Jun 2017 16:03:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1497470624; bh=s9Sr008aMezrJqsquW9Z4dHjpknZnKp+HOdyxRgClno=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From:Subject; b=edA2tWjTQrv2nMDW7dMPlrNJmttyLA+qg8OVu6cXx0SmWGvlPyzhS4b7yoHYemFfR1wi5OBpdEtJrW
ecym/riXT6y9BbDEDD+ysT7iCZ3lqIIsIR/h0+Adb/jLtAJUBEQEikRyTGfMo5Eai+7JrdkGEXP9IM/vVmqnLUxPh4Gj5qZOeCW309kBX9bIaDHUpN74ZhzfOxTraoXAsfqLUa21YsBXa7STTucBr80XEHp8w7zf3ouaIuAyisNm0mRmTuODb2Dk4qO2OwVstLJpVtdZ+GsladEttZlXUs18zjSPWmy1OJZue5JsU9inBG3t/9oMZKUlF8YO0d7IgGkvfm0w==
Received: from [98.138.101.131] by nm10.bullet.mail.ne1.yahoo.com with NNFMP; 14 Jun 2017 20:03:44 -0000
Received: from [98.138.226.30] by tm19.bullet.mail.ne1.yahoo.com with NNFMP; 14 Jun 2017 20:03:44 -0000
Received: from [127.0.0.1] by smtp201.mail.ne1.yahoo.com with NNFMP; 14 Jun 2017 20:03:44 -0000
X-Yahoo-Newman-Id: [e-mail]
X-Yahoo-Newman-Property: ymail-3
X-YMail-OSG: A5TTZBcVM1lZ9isS7G4nHKl4JSi.MLUsEIAmzhIl7EpAIeS
 2waHuoYmPm9DxVBeWp_H_oefZbtkZYrMyIrVHNpmxr2zj24bLog7p5K5Kj9S
 uB4QvRxC.whGtONDx0sBSKw3LDi_VZNXhHBofY.EUAe9H6rDQGU7t6U2snMg
 iOKmlWkVpCcJiCEKOhOCKb_bNzfDhhqC1lYitaddBYQvJLqm_aO66IxBAZrA
 eM3jGjx7mGK4fVJsfQuGyqt5etTClr8FNElmpLZskQ2jtbhDpGtjXHGRXrpK
 BgyqlaVoBk5iLVxf_WwTz2QhMihgzPw7OZTNniDj9aqsWoe.Q0BMvlubaxlr
 lsS7YGVcqaTTXA_tyOlSEI0O6jT_7cWKv.0iPfbvNwFA0m9TK8J.3SU8f1fC
 ju7HaWOXSvmiLj.kvS_S28SPrmJMPoV3hJhETBa0K6xBjd4ZyeioRvuAG1zH
 2eLbG4J6IZS8E3IOdbBCovZnxac1SCEi2dxbfY3B0P3YB5HPFqD5jHxoITir
 899hgggOKLczHlQSKXoTaGrXBgkZqu21U5he0LBm_IzJMSkFNQA5g52nU
X-Yahoo-SMTP: OIJXglSswBDfgLtXluJ6wiAYv6_cnw--
Subject: Re: [PATCH 08/27] VFS: Introduce the structs and doc for a filesystem
 context [ver #5]
To:     David Howells <[e-mail]>, [e-mail],
        [e-mail]
Cc:     [e-mail], [e-mail],
        [e-mail],
        [e-mail],
        [e-mail]
References: <[e-mail]>
 <[e-mail]>
From:   Casey Schaufler <[e-mail]>
Message-ID: <[e-mail]>
Date:   Wed, 14 Jun 2017 13:03:43 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.1.1
MIME-Version: 1.0
In-Reply-To: <[e-mail]>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Sender: [e-mail]
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: [e-mail]
X-Virus-Scanned: Maia Mailguard 1.0.4

Based on this data, I assume "from" in syslog is equal the sender address from which the mail was delivered to me and from in database (metadata) is the real writer of the mail.

For not getting confused while creating rules, I propose to clarify this issue.

[Janos SUTO]

The mail header 'From' address should be logged.

[Uwe Kiewel]

agreed, it should.

Well, it seems to to be working for mailing lists. Check example above: Header and metadata in v_messages contains an other from als the logged from.

[Janos SUTO]

Well, I'd like to correct myself. The logged from entry is the envelope sender you can see in the MAIL FROM: smtp command. Usually it's the same as in the mail header From: address. However as you have observed for some mailing lists it's a list address. I'm not sure if piler needs any fix.

[Uwe Kiewel]

OK, but I think it is not only a logging issue.

In that case, my recipeint address is [e-mail]. BUT, this recipient address appears in no relevant header data. So, logging in to piler with this account I do not see any mail from the mailing list. I see only the mail for subscribing and confirming and so on which are send directly to me. Of course, it is only related to (some) mailing lists and the question is: Does it make sense to archive mails from a mailing list.

I think about posting a note to this issue on the piler-user mailing list to see how others are thinking about.

[Janos SUTO]

OK, do so.

[Janos SUTO]

The problem you mentioned on the mailing list occurs on external mailing lists that the piler gui can't query for membership info. In this case I'd solve the issue with the groups feature, ie. add the mailing list address as an extra email addresses to the given user. Or you may do the same with a post auth hook, and append the mailing list address to the user's addresses.

[Janos SUTO]

I didn't see many responses on the mailing list, so I suggest to keep piler as it is, and discard emails from (external) mailing lists with archiving rules.