HTTPS SSH

SPECS monitoring: The Event Hub

This repository contains the SPECS Event Hub monitoring module. The general architecture of the Event Hub can be found here.

Table Of Contents

  1. Installing the Event Hub

  2. Configuring the Event Hub

    1. Core configuration
      1. Handlers configuration
      2. Archiving configuration
  3. Using the Event Hub

    1. Starting the Hub
    2. Sending messages
    3. Receiving events
  4. SPECS messages

    1. Input and Event message
    2. Event Hub's internal message
  5. SPECS monitoring plugins

    1. Decoders
    2. Filters

Installing the Event Hub

The Event Hub requires a Go work environment to be setup for the binary to be built. Prior to installing the Hub you must install and configure Go (the GOPATH and GOROOT environment variables must also be set).

The Hub is implemented using Mozilla Heka version 0.9.2 and therefore requires Heka to be setup for building the binary. However, you will not need to install Heka as this task is automated for you by the Hub's build process. Because the Hub is based on Heka, all Heka prerequisites must be available. Please check Heka's installation instructions for its list of prerequisites.

Once you have all prerequisites, you can follow with the Event Hub's installation. For this, all you need is the install.sh script in this repository. You do not need to download anything else. Copy the install.sh somewhere on your computer and run it. Upon its successful execution, in the same folder where you executed the script you should find an executable file named event-hub.

The Downloads section of this repository contains two built versions of the Event Hub, one executable on Microsoft Windows systems and one for Linux systems. Both have been built for 64-bits architectures.

Configuring the Event Hub

The Event Hub configuration file specifies how to handle received messages and requests for event notifications, and also what Sieve Filters to load (check the Sieve Filter section for more details on it). The configuration file is in TOML format.

The configuration file is broken into sections. There is a section named hekad which contains global configuration options described here. To the configurations options described there, the Event Hub adds three others, described in the next section.

Besides the global configuration section, the configuration file will contain a section for each instance of a plugin which must be loaded by the Event Hub. The section name specifies the name of the plugin, and the “type” parameter specifies the plugin type. For the Event Hub the only accepted plugin types are SieveFilter, SpecsCounterFilter, SpecsDecoder and SpecsRestDecoder. The configuration options for each of these plugins are presented in their respective sections.

Core configuration

The Event Hub's core configuration extends the standard Heka's global configuration. The Hub has three related configuration options and an optional configuration option for specifying the connection details for the Event Archiver:

  • buffer_size (int, optional): The Event Hub temporarily stores events in buffers before sending them to the interested clients. This option specifies the size of these buffers. Defaults to 1024.
  • handlers_configuration (string, required): This is a base64 encoded string which specifies how input messages and outputs should be handled.
  • port (int, optional): The port on which the Event Hub will receive HTTP requests. Defaults to 8080.
  • archiver: A configuration section containing the connection details for the Event Archiver. For details, see Section Archiving configuration.

Handlers configuration

Clients can send input messages to the Event Hub using HTTP. The output events, routed through the Hub, can be received by interested clients also using HTTP. The Hub uses the handler_configuration option in order to determine which HTTP requests are sending input messages to the Hub and which are actually requesting events from the Hub. As such, a handler is composed of a set of parameters as follows:

  • path_pattern: a regular expression for the HTTP request URI path. This path will be checked against the path of the received request and, in case of a match, the rest of the parameters of the handler will specify how to handle the request. This is a mandatory parameter.
  • action: this will indicate if this is an input message or a request for receiving events from the Hub. This is a mandatory parameter. The only possible values are input and stream_output.
  • method: an HTTP method. For now, this should be POST when sending input messages and GET when requesting messages from the Hub. This is a mandatory parameter.
  • path_pattern_stream_group: an integer value indicating which group from the regular expression in the first parameters contains a stream name. For more details on this parameter see section Receiving events. This is mandatory when the action parameter is stream_output.
  • decoder: the decoder used for decoding input messages. This should correspond to the name of a Heka decoder plugin. The only plugins which can be used here are the SpecsDecoder and the SpecsRestDecoder. This option is mandatory when the action parameter is input and is mutually exclusive with the encoder parameter.
  • encoder: the encoder used for encoding events. The only encoders to be used are SpecsEventEncoder and SpecsRestEncoder. This option is mandatory when the action parameter is stream_output and is mutually exclusive with the decoder parameter.
  • annotations: a key-value mapping. This is will be used by the SpecsRestDecoder and SpecsRestEncoder.

The base64 decoded handlers must comply to the following syntax:

handlers.<index>.path_pattern = <regular expression describing a HTTP request URI path>
handlers.<index>.action = input|stream_output
handlers.<index>.method = POST|GET
handlers.<index>.path_pattern_stream_group = <integer value>
handlers.<index>.decoder = SpecsDecoder
handlers.<index>.encoder = SpecsEventEncoder
handlers.<index>.annotations = <json object>

The <index> placeholder is an integer value used for grouping and identifying the parameters of a single handler. As mentioned above, a handler will not have all the parameters in the syntax description above, but only those related to its associated action.

The annotations parameter is a simple JSON object, containing only string values.

If the annotations parameters belongs to a handler which has a SpecsRestDecoder associated with it, the Hub can handle only attributes with prefix specs-decoder-rest: and values starting with one of the prefixes http:path:group: or http:headers:. In this case, what follows after the prefix of the attribute is interpreted as a field of a Specs Event (see Section Input and Event message) and what follows after the prefix of the value is interpreted as the a way to get the value of that field.

If the prefix of the value is http:path:group:, then the value of the field will be determined from the request path, using the path_pattern parameter of the handler. In this case, after the prefix, an integer value is expected representing the group in the path_pattern regular expression which, after matching, will contain the searched value of the field.

If the prefix of the value is http:headers:, then the value of the field will be taken from the HTTP request header with the name given by what follows after the value of JSON attribute.

If the annotations parameters belongs to a handler which has a SpecsRestEncoder associated with it, the Hub can handle only attributes with prefix specs-encoder-rest:. In this case, what follows after the prefix of the attribute is interpreted as a field of a Specs Event (see Section Input and Event message) and the value associated with the prefixed attribute will be used for encoding the event before sending it to the client. Using the annotations parameter, the SpecsRestEncoder will encode an event from the Hub's internal format to a JSON object, where the keys in the JSON object are given by the values in the annotations parameter and the values are taken from the corresponding fields of the Specs event. This way, a client may "translate" the Specs event format into something more appropriate for its needs.

The example below defines four handlers, two for input and two for output streaming. The handler with index 0 is an input handler with an associated request URI path starting with /events/. The handler with index 1 is an output handler with an associated request URI path starting with /streams/. The handler gives 1 as the path_pattern_stream_group which means that whatever the client will put in the request URI path between /stream/ and /consume will be interpreted as a stream name and only events marked with that stream will be sent to the client. The handler with index 2 is an input handler using a SpecsRestDecoder. The annotations parameter of the handler specifies the rules based on which the decoder will populate the fields of the Event Hub's internal message:

  • "specs-decoder-rest:component" : "http:path:group:1": means that the component will be taken from the request path URL based on pattern matching. The component will be in the first group in the matched regular expression given by the path_pattern parameter. For example if the request path is /events/component1/object1, the component field will be component1;
  • "specs-decoder-rest:content-type" : "http:headers:content-type": means that the content-type of the message is given in the content-type field of the header of the HTTP request.

The handler with index 3 is an input handler using a SpecsRestEncoder. The annotations parameter of the handler specifies the rules based on which the encoder will build the output JSON object to send to the client. It build a JSON object starting from an Specs event as defined in Section Input and Event message, but this output object will have only 4 key-value pairs:

  • "specs-component" with the value given by the component attribute of the event;
  • "specs-object" with the value given by the object attribute of the event;
  • "specs-data" with the value given by the data attribute of the event;
  • "specs-token" with the value given by the token attribute of the event.

Example handlers configuration:

handlers.0.path_pattern = ^/events/([^/]+)$
handlers.0.action = input
handlers.0.method = POST
handlers.0.decoder = SpecsDecoder

handlers.1.path_pattern = ^/streams/([^/]+)/consume$
handlers.1.action = stream-output
handlers.1.path_pattern_stream_group = 1
handlers.1.encoder = SpecsEventEncoder

handlers.2.path_pattern = ^/restevents/([^/]+)/([^/]+)$
handlers.2.action = input
handlers.2.method = "POST"
handlers.2.annotations = {
    "specs-decoder-rest:component" : "http:path:group:1",
    "specs-decoder-rest:object" : "http:path:group:2",
    "specs-decoder-rest:content-type" : "http:headers:content-type"
}
handlers.2.decoder = SpecsRestDecoder

handlers.3.path_pattern = ^/streams/([^/]+)/restconsume$
handlers.3.action = stream-output
handlers.3.path_pattern_stream_group = 1
handlers.3.encoder = SpecsRestEncoder
handlers.3.annotations = {
    "specs-encoder-rest:component" : "specs-component",
    "specs-encoder-rest:object" : "specs-object",
    "specs-encoder-rest:data" : "specs-data",
    "specs-encoder-rest:token" : "specs-token"
}

Archiving configuration

The Event Hub will send, if configured so, all received events to the Event Archiver. The Event Archiver provides an HTTP interface which will be used by the Event Hub to store the events.

The connection details of the Archiver must be described in the archiver section of the core configuration section of the Event Hub. If the section is missing from the configuration file, the Event Hub will not archive any event.

The archiver section may contain the following options:

  • address (string, mandatory): URL address of Event Archiver to which events should be sent. Must begin with “http://” or “https://”.
  • http_timeout (int, optional): time in milliseconds to wait for a response from the Archiver. Defaults to 0 (no timeout).
  • username: if specified, HTTP Basic Auth will be used with the provided user name.
  • password: if specified, HTTP Basic Auth will be used with the provided password.
  • tls: A sub-section that specifies the settings to be used for any SSL/TLS encryption. This will only have any impact if an “https://” address is used. Details about this subsection can be found in the official Heka documentation, here.

Example Archiving configuration:

[hekad.archiver]
    address = "https://localhost:8086"
    [hekad.archiver.tls]
        cert_file = "/usr/share/heka/tls/cert.pem"
        key_file = "/usr/share/heka/tls/cert.key"
        client_auth = "RequireAndVerifyClientCert"
        prefer_server_ciphers = true
        min_version = "TLS11"

Using the Event Hub

Starting the Hub

You can start an Event Hub process based on a certain configuration by using the command:

event-hub -config=<toml file>

Sending messages

In order to send a message to the Event Hub, a client should send a HTTP POST request with a path matching one of the Hub's input handlers path pattern. The request should contain a SPECS input message conforming to the format described in section Input message.

For example, if the Hub is configured with the handlers in the example in section Core configuration and listens on the endpoint localhost:8080, a valid request URI would be http://localhost:8080/events/whatever.

Receiving events

A client that wants to receive events from the Event Hub, it must send a HTTP GET request with a path matching one of the Hub's output handlers path pattern.

For example, if the Hub is configured with the handlers in the example in section Core configuration and listens on the endpoint localhost:8080, a valid request URI would be http://localhost:8080/streams/specs/consume. Once the connection is established, the Event Hub will start streaming events to the client, where all events are marked with the stream specs. The format of the output event is the same as the input message.

Each event has a token attribute which uniquely identifies the message. This token can be used by the client when requesting events. To understand better how it can be used, lets take the following scenario. The client connects to the Hub using the above URI and starts receiving events from the Hub. During the streaming, something happens and the connection is dropped. When the client reconnects, it will want to receive only the events generated after the connection was dropped. To be able to handle this scenario, the Hub keeps a buffer of the latest messages for each stream. Each buffer has an associated UUID. Each event in the buffer has a token composed from this UUID and a sequence number. This token is sent by the Hub to the client, in the event. The events are sent ordered by sequence number. When the connection between client and Hub drops, the client knows which was the last event it received and all it has to do is to send the token in this event to the Hub when it reconnects. The serialized token can be send in the SPECS-Heka-HTTP-Gateway-Stream-Sequence request header. By serialized token we understand here the token in the form UUID:sequence_number. If the Hub receives this header and it was not restarted in the meantime, it will send only events with sequence number greater than the one in the token. If it was restarted, it will send all events in the respective stream buffer.

SPECS messages

Input and Event message

The monitoring events are received by the Event Hub as JSON messages. An input message received by the Hub may contain several such JSON messages. The events generated as output by the Hub will follow the same JSON format, unless the SpecsRestEncoder is used.

The JSON message contains the following attributes:

  • component (string): the unique identifier (usually a UUID) of the component instance which generated the event, i.e. a virtual machine, a RabbitMQ instance or a web-server instance
  • object (string): is a hierarchical dot-notation string that pinpoints more accurately the event source within the 'component', i.e. queue, exchange, etc.
  • labels (array of strings): a list of hierarchical strings that provides a way to give a context to the event, i.e. one such label could be vm, to denote all events coming from a VM, or user-x, to denote all events belonging to a certain user
  • type (string): an hierarchical string indicating what type of event is this, like "syslog", "structured-log", "cloudwatch.metrics"
  • data (JSON object): depends on the type of event
  • timestamp (number): time of the event, in seconds
  • token (JSON object): the token identifying an event generated by the Event Hub. The object has two attributes uuid, containing the UUID in the token, and seq, containing the sequence number in the token.

Event Hub's internal message

The input message is decoded by the decoder associated with the Hub's selected handler into a custom Heka message which will be called from now on internal message. Each JSON message in the input message will be decoded into an internal message. For the internal messages the type message variable of the Heka messages will be set to specs.monitoring.event and the payload message variable will contain the input JSON message. For each attribute in the input JSON message, the internal message contains a field variable. Thus, the message contains the following fields (in brackets is the Go type of the field):

  • component (string)
  • object (string)
  • labels ([]string)
  • type (string)
  • data (interface{})
  • timestamp (float64)
  • token (a custom struct type)

SPECS monitoring plugins

This section describes the Heka plugins developed for the Event Hub. For more details on each kind of Heka plugin, please check the official Heka documentation.

Decoders

SpecsDecoder

Parses a payload containing one or more JSON input messages and builds an Event Hub's internal message for each JSON message.

If your Event Hub handler_configuration contains a reference to this decoder, the Hub will automatically configure and start an instance of the decoder for you.

But you may start additional decoders using your configuration file.

Example Heka configuration:

[TheSpecsDecoder]
type = "SpecsDecoder"

SpecsRestDecoder

Parses an HTTP request received by the Event Hub and builds an Event Hub's internal message based on the request. The request must contain the data attribute in its body. The fields of the message are populated based on rules defined in the handler corresponding to the request path. These rules are defined in the annotations parameter of the handler. Unlike the SpecsDecoder, this decoder can handle only a single message in each request.

If your Event Hub handler_configuration contains a reference to this decoder, the Hub will automatically configure and start an instance of the decoder for you.

But you may start additional decoders using your configuration file.

Example Heka configuration:

[TheSpecsRestDecoder]
type = "SpecsRestDecoder"

Filters

SieveFilter

This filter adds a field called stream to all Heka messages it receives. This field is used in the Event Hub for further routing of messages to specific outputs. If the configuration used for starting Heka contains more than one SieveFilter, each filter will create a copy of the received message in which will add the stream field giving it the value of the filter's stream value. If the received message already has the field stream, then it will replace its value. The copied message will be injected back in the Hub.

During configuration of the filter, one should configure the message_matcher option such that the filter will not receive its own messages. Specifically, if the configured stream for a filter is "abc", then its message_matcher must contain Fields[stream] != 'abc' || Fields[stream] == NIL

Config:

Besides the configuration options that are universally available to all Heka filter plugins, this plugin also supports the following configuration options:

  • message_matcher (string, optional): This is a standard option, but we added it in this description in order to specify its default value: Fields[stream] != NIL.
  • stream (string, required): Specifies the value of the stream field to be appended to the messages.

Example Heka configuration:

[TheSieveFilter]
type = "SpecsSieveFilter"
message_matcher = "Fields[object] == 'exchange.queue1' && (Fields[stream] != 'demo' || Fields[stream] == NIL)"
stream_id = "demo"

SpecsCounterFilter

Once per ticker interval a SpecsCounterFilter will generate one or more Event Hub's internal messages. If the group_messages_by configuration option was left empty, the filter will generate a single message indicating the number of messages that matched the filter’s message_matcher value during that interval (i.e. it counts the messages the plugin received). If the group_messages_by configuration option was set, then the filter will group the messages received by the plugin based on the value of the field in the group_messages_by option, and once per ticker interval will generate one message for each such value, counting how many messages were received for each value of the grouping field encountered.

The generated internal messages will have the Heka type specs.counter.v1.

The label field of the message will contain a single string with the value of the field used for grouping messages, i.e. component names, object names, event types.

The data field will have the structure:

Operator string
Result   string

Config:

Besides the configuration options that are universally available to all Heka filter plugins, this plugin also supports the following configuration options:

  • group_messages_by (string, optional): Specifies based on which input message field to group messages for counting. Valid values are: component, object and type. If the option is missing or is the empty string all SPECS messages in a ticker interval will be counted.
  • message_matcher (string, optional): This is a standard option, but we added it in this description in order to specify its default value: Type != 'specs.counter.v1'.
  • ticker_interval (int, optional): Interval between generated counter messages, in seconds. Defaults to 300. (This is a standard option, but we added it in this description in order to specify its default value).

Example Heka configuration:

[TheSpecsCounterFilter]
type = "SpecsCounterFilter"
message_matcher = "Fields[component] != NIL && Type != 'specs.counter.v1'"
ticker_interval = 10
group_messages_by="component"

Example Event Hub's internal output message payloads:

{
    "component":"specs-counter-filter",
    "object":"",
    "labels":["component1"],
    "type":"specs.aggregation",
    "data":{
        "operator":"count",
        "result":"2"
    },
    "timestamp":1414701485
}

{
    "component":"specs-counter-filter",
    "object":"",
    "labels":["component2"],
    "type":"specs.aggregation",
    "data":{
        "operator":"count",
        "result":"2"
    },
    "timestamp":1414701485
}