File content monitoring similar to tail -f [feature request]

[Don]

Could you consider adding monitoring for content added to the file.

There is already similar functionality in rsyslog (imfile + ommail), but, it's monstrously complicated to configure it correctly, rsyslogd is not ubiquitous, rsyslog configuration language is changing greatly between versions, and finally, rsyslogd have disadvantage - when such monitoring is first configured, it starts from parsing whole file from beginning sending emails for all found lines, which could be a lot of garbage, because only new entries are interesting. Additionally, monit have built-in actions functionality which rsyslog does not have.

• Why Monit and not just [bash] script in cron? Because between script runs file position needs to be remembered, and remembering it complicates the script. Otherwise, daemon should be coded which complicates things again (robust daemon is complicated coding). Monit is already daemon, and thus it does not need to save file position, it could be keept just in memory because of tail -f nature.

• If you going to implement it, please implement all three modes of ReadMode of rsyslog, because they all are useful:

• line mode is good for monitoring for bad lines, like all new lines with 'ERROR' substring.

• multi-line paragraph and indented mode is useful for monitoring of other scripts error messages. For example, I could redirect output of some lame ruby script to log file, where script normally should not output anything, but, on error - whole error with traceback could be mailed to admin. In this use case regexp matching is not useful, but, it could be for someone. rsyslog v8 also have startmsg.regex to replace multi-line readMode.

• Additional useful functionality which monit could have: state triggering lines. If particular regexp is found, this could be interpreted as bad state until some other regexp is found, which will trigger good state (and alert that 'test is succeeded'). Possible use case:

Exim periodically log number of active connections like this: SMTP connection from [11.22.33.44]:50049 (TCP/IP connection count = 131)

when amount of connections approach smtp_accept_max it will stop accepting new mail, which is problem and admin could be alerted.

There could be configured regexp to trigger 'bad state' and alert (or some repair script), and another regexp to trigger 'good state' when connection count became low. Of course connection numbers should be matched with regexps.

[Tildeslash]

Hello,

the file content monitoring very similar to the description is implemented already: https://mmonit.com/monit/documentation/monit.html#FILE-CONTENT-TESTING

[Don]

Thanks. But note, that I described more functionality, in case you want to enhance.

[Tildeslash]

Thank you. Suggestions are always welcome, especially suggestions that have been thought through. Monit already supports line based monitoring with regex as mentioned. Multi-line testing should be supported IMO. In fact, the file content check code in Monit could benefit from refactoring and use a more "advanced" buffer handling via mmap. This would also make it trivially to support multi-line matching. It should be noted that regular expressions does not easily lend itself to multi-line matching, though it is certainly possible for simpler cases.

State matching as you suggest, I feel, is out of scope for Monit. Changing status of this issue to on hold with the intent of supporting multi-line matching.

In addition, if a match occurs, Monit should report several lines before and after the matched line, for context. This is particularly useful with exceptions stack traces which could be long and require context to understand.

Ref: d0649dd

[Don]

Thanks for reply.

I tried content matching on monit 5.14 from centos 7 from epel repo, and, it seems, that version not support content matching yet (monit reload says syntax error 'content'). This leads me to suggesting -- maybe, you could state in the docs since which version this or that feature is supported?

If context will be configurable this will be good, because context is not always meaningful. For example I want to monitor backup log for just /ERROR/ lines and send these alerts to client who could be confused with extra text, if it's not relevant.

State detection is not really hard to be excluded from monit -- 1) first state is unknown, and monit does not send anything, if monit is restarted (not reloaded) in the middle of other state, state is unknown; 2) then just two regexps could determine failed or normal state with appropriate message. Doesn't look too hard, and opens more possibilities for monitoring. Currently, with per-line content alerts we can not alert on just bad lines because bad lines, if they are frequent, (and they are frequent for Exim) will trigger too much alerts.

Btw, you use syntax for content matching = "regexp", but I would suggest to use modern standard for regexp matching syntax (which is used awk, perl, ruby, and other languages) =~ /regexp/ that ways people will easier comprehend it. Or maybe two syntaxes for backward compatibility. =~ /regexp/ also would suggest this is regexp and not just substring matching.

Btw2, you also have confusing example for content matching:

  check file syslog with path /var/log/syslog
        ignore content = "^monit"
        if content = "^mrcoffee" then alert

There can not be string starting from 'monit' or 'mrcoffee' in syslog, unless we (users) don't understand something about content matching in monit.

"^" is good to show these are regexps, though.

[Tildeslash]

The content test is present in monit for very long time (i think version 3.x), but original syntax was "IF MATCH", which was renamed in monit 5.16 to "IF CONTENT" as part of cleanup and to unify the syntax (the content test is also supported in http protocol test).

The online manual is always for the latest Monit version, each release also bundles the manual for the given version. Changes are described in release notes: https://mmonit.com/monit/changes/

Thanks for the invalid example notification, fixed: https://bitbucket.org/tildeslash/monit/commits/2e8d339a1a8e/

[Don]

Thanks for pointing out if match!

[Pavel Vasev]

Hi guys! What about multi-line logs? As for me, we have rails production.log, wherever each "log entry" consist of multiple lines, and we want to see all those lines of each log entry with error flags in our email..