Custom alert channels

[Don]

Sometimes, it's useful not to send alert via email, but to act directly on service state, for example send SMS or instant message. With exec action it is not achievable, because, it only acts on fault and not on service restoration. Thinking about this problem, I thought it would be useful to have special URLs in place of emails to use as alert targets.

For example, to define alert exec:/root/sms-script.py and when any event is happened, instead of sending email, script /root/sms-script.py is executed with MONIT_ env variables.

This will spare people from writing email2sms, email2whatsever gateways. Also, this remove complicated SMTP system from alert path. And will simplify coding, as no need to parse message, but just use well defined env variables. (You could also add env variable to signal this is failure or restoration event.)

[Tildeslash]

Sending alert for recovery is possible, you just need to add the optional "else" part:

if failed <test> then exec "..." else if succeeded then exec "..."

We also plan to add custom alert channels, see here for more details: https://bitbucket.org/tildeslash/monit/issues/134/patch-added-support-for-local-usr-sbin

[Tildeslash]

putting on hold (will reopen when implementation will begin)

[Don]

Thanks much! I will try that.

[Don]

Hmm. After testing, it seems that exec action is executed every cycle. Was there a bug in monit 5.14 or I missed something? man monit on the system doesn't state that "[t]he program is by default executed only once, on the state change" unlike last web doc does.

Test is like this:

check host xx.xx with address x.x.x.x
  if failed port 80 protocol http for 3 cycles then exec /root/send-sms.rb
    else if succeeded then exec /root/send-sms.rb

[Tildeslash]

Please upgrade monit ... the exec action was changed in monit 5.16.

[Tildeslash]

Issue #534 was marked as a duplicate of this issue.

[Tildeslash]

Issue #925 was marked as a duplicate of this issue.

[Karthik T]

@Tildeslash I am not clear on the status here. Is it something that Will not be done? Accepting PRs? Waiting to be done? Any reasons to the status?

[Tildeslash]

@Karthik T we plan to do it, but no ETA is available

[Karthik T]

Given this has been waiting for 6 years.. would you be open to a PR replicating https://bitbucket.org/tildeslash/monit/pull-requests/19/support-for-alerting-by-shell-commands/diff ?

[Tildeslash]

@Karthik T no. We want to make generic interface that’ll allow you to use choose the alert channel for each test

[Danny Weldon]

It would be nice to also have several alert levels such as Warning to provide compatibility with Nagios checks. It may be necessary to provide the ability to not alert on warnings in case that was desired, but I have no idea if this would over-complicate the config syntax and code so a simple solution would be to send all alerts so if a user doesn’t want alerts for warnings, they just don’t configure them. Or perhaps a warn channel could be set to a different notification type.

Also to be technically fully correct, there could technically be four levels: Critical, Major, Minor, Warning. I’m not suggesting that they be implemented, but kept in mind in case it was desired down the track, so that any config syntax enhancements would still be accommodating.