Very cool :-) We'll keep this PR open, if okay with you, but we are hesitant to merge. The reason is that IOHO this type of integration belongs on a higher abstraction level. For instance, something like,
set slack "/path/to/slack-alert"
if failed X thenexec slack
Where slack-alert is a script or a program that send alerts/events to slack. This way, it will be easier to support third-party system without having to modify Monit. For instance, rrdtool, Graphite, Nagios, PagerDuty etc etc. If we include code in Monit for every third-party system we would like to integrate with it can quickly become a maintenance problem.
What i could easily do to make it more generic is to support a more generic http+json message. I would need a small template library to limit the maintenance prb. It could also be used in other part of the code i think (for the xml generated for mmonit or to change the default template returned by the builtin webserver).
A template system is a solution that could scale and to some extent be used, but not quite. In M/Monit we use a little bit of clearsilver as a template system and it is tempting to use it in Monit as well to support generating JSON, XML, HTML, Plaintext SMTP alerts etc. However, to POST data to a third-party system such as slack you still need to adhere to their API and auth system. If they do any changes in the API it will require changes in Monit and recompile which is brittle and can break between versions. So, IOHO, it might be more flexible to do integration to third-parties outside of Monit via for instance an exec call as hinted at above. Monit provides the following environment variables to the program being executed. If that is not enough, Monit also provides a read-only "API" which can be used by a program to get data off Monit via http://localhost:2812/_status?format=txt where format can be text or xml. Adding JSON format as a third option to this API is trivial.
I would love to see this as a the generic http-json notifier, and I was surprised it wasn't core to Monit already (given the integration with m/monit) Analogous to sending mail, you configure it for event types, but it's not part of each service. If it gets configured, like mail, it gets triggered on each action...if that makes sense. I personally would like to send my notifications to Hubot (which then would send them on to Slack). It'd be trivial to create a notifier plugin to Hubot that you could configure to drop/process different event types.
I am working around this now by having a script process the monit log, which adds a minute or so of delay to each notification.
I converted all alerts in the exec script. Is there a way to trigger the script when f.e. a resource limit check succeeded?
I can't find a way to trigger this exec overtime an alert was triggered (when I had mail alerts)
setting exec action on success is possible as well, although it'll look little bit complicated:
if failed XYZ then exec ... else if succeeded then exec ...
The [else if succeeded ...] part is optional and if not used, expands to [else if succeeded then alert], but you can override it.
Note: we have also refactored the curl based example into ruby script (http://mmonit.com/wiki/MMonit/SlackNotification) ... the MONIT_* environment variables (especially MONIT_DESCRIPTION) may contain some characters which need to be JSON-escaped - ruby allows simple JSON escaping (unlike cURL).
Tildeslash The problem with the suggested solution are the default alerts sent by actions. So if you want to restart a service that no longer exists and send a slack notification you need to do the following:
if does not exist then exec "/path/to/slack-script.sh"
if does not exist then start
But both the "exec" action and the "start" action send alerts, meaning you would receive two email alerts. Obviously one can disable alerts for that service, but then you wouldn't get one, which also isn't the desired behaviour. Being able to hook into the alert system in some way would be really helpful I feel.