Add support for notifying via

#15 Declined
  1. Pierre Aubert

Hello, this patch allow Monit to push notif to

The code is modeled after mmonit trigger and follow the same logic.

Comments (10)

  1. Tildeslash repo owner

    Very cool :-) We'll keep this PR open, if okay with you, but we are hesitant to merge. The reason is that IOHO this type of integration belongs on a higher abstraction level. For instance, something like,

    set slack "/path/to/slack-alert"
    if failed X then exec slack 

    Where slack-alert is a script or a program that send alerts/events to slack. This way, it will be easier to support third-party system without having to modify Monit. For instance, rrdtool, Graphite, Nagios, PagerDuty etc etc. If we include code in Monit for every third-party system we would like to integrate with it can quickly become a maintenance problem.

  2. Pierre Aubert author

    What i could easily do to make it more generic is to support a more generic http+json message. I would need a small template library to limit the maintenance prb. It could also be used in other part of the code i think (for the xml generated for mmonit or to change the default template returned by the builtin webserver).

    1. Tildeslash repo owner

      A template system is a solution that could scale and to some extent be used, but not quite. In M/Monit we use a little bit of clearsilver as a template system and it is tempting to use it in Monit as well to support generating JSON, XML, HTML, Plaintext SMTP alerts etc. However, to POST data to a third-party system such as slack you still need to adhere to their API and auth system. If they do any changes in the API it will require changes in Monit and recompile which is brittle and can break between versions. So, IOHO, it might be more flexible to do integration to third-parties outside of Monit via for instance an exec call as hinted at above. Monit provides the following environment variables to the program being executed. If that is not enough, Monit also provides a read-only "API" which can be used by a program to get data off Monit via http://localhost:2812/_status?format=txt where format can be text or xml. Adding JSON format as a third option to this API is trivial.

  3. Chris Stolfi

    +1 for adding JSON to /_status

    I would love to see this as a the generic http-json notifier, and I was surprised it wasn't core to Monit already (given the integration with m/monit) Analogous to sending mail, you configure it for event types, but it's not part of each service. If it gets configured, like mail, it gets triggered on each action...if that makes sense. I personally would like to send my notifications to Hubot (which then would send them on to Slack). It'd be trivial to create a notifier plugin to Hubot that you could configure to drop/process different event types.

    I am working around this now by having a script process the monit log, which adds a minute or so of delay to each notification.

  4. jverdeyen

    Any update on this very nice feature? Or is there already a .deb file available for Ubuntu 14.04? +1 Thanks!

  5. Tildeslash repo owner

    As noted in the previous update, we prefer more general notification setup, which will allow simple integration of any 3rd party service.

    In the meantime we have prepared instruction for how to setup Slack notification for both Monit and M/Monit using exec action:

  6. jverdeyen

    @tildeslash wow, super! Thanks! I already compiled this pull request, but the solution you propose is much more versatile. Thanks!

  7. jverdeyen

    I converted all alerts in the exec script. Is there a way to trigger the script when f.e. a resource limit check succeeded? I can't find a way to trigger this exec overtime an alert was triggered (when I had mail alerts)

  8. Tildeslash repo owner

    setting exec action on success is possible as well, although it'll look little bit complicated:

    if failed XYZ then exec ... else if succeeded then exec ...

    The [else if succeeded ...] part is optional and if not used, expands to [else if succeeded then alert], but you can override it.

    Note: we have also refactored the curl based example into ruby script ( ... the MONIT_* environment variables (especially MONIT_DESCRIPTION) may contain some characters which need to be JSON-escaped - ruby allows simple JSON escaping (unlike cURL).

  9. Leon Aves

    @tildeslash The problem with the suggested solution are the default alerts sent by actions. So if you want to restart a service that no longer exists and send a slack notification you need to do the following:

    if does not exist then exec "/path/to/"
    if does not exist then start

    But both the "exec" action and the "start" action send alerts, meaning you would receive two email alerts. Obviously one can disable alerts for that service, but then you wouldn't get one, which also isn't the desired behaviour. Being able to hook into the alert system in some way would be really helpful I feel.