HTTPS SSH

SPECS Monitoring -- Event Archiver

SPECS Event Archiver is in charge with SPECS Monitoring Events data archival and retrieval through a public Web REST interface.

Table of contents

Installation

Requirements

  • Flask
  • MongoDB
  • Flask-PyMongo

Installation steps

These installation steps are compatible with mOS 4.0.x or OpenSUSE 13.1 (with SPECS repositories activated) environments.

  • Install the requirements
zypper install python python-pip mercurial
cd /opt
hg clone https://bitbucket.org/silviu001/specs-monitoring-event-archiver
cd specs-monitoring-event-archiver
pip install -r requirements.txt
cd /root
zypper addrepo --no-gpgcheck https://repo.mongodb.org/zypper/suse/11/mongodb-org/3.0/x86_64/ mongodb
zypper install mongodb-org
  • Start the services
service mongod start
/opt/specs-monitoring-event-archiver/bootstrap.sh start
  • Access the REST interface
curl http://localhost:10101/

Service configuration

SPECS Monitoring Event Archiver can be customized by editing the etc/vars.sh script:

  • SPECS_MONITORING_EVENT_ARCHIVER_ENDPOINT_IP='0.0.0.0'
  • the IP address the REST service should listen to;
  • SPECS_MONITORING_EVENT_ARCHIVER_ENDPOINT_PORT='10101'
  • the Port number the REST service should bind to;
  • SPECS_MONITORING_EVENT_ARCHIVER_DEBUG_MODE=0
  • activate the debug mode: 0 disabled | 1 enabled;
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_IP='127.0.0.1'
  • MongoDB database IP address;
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_PORT='27017'
  • MongoDB database Port number;
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_DBNAME='monitoring'
  • MongoDB database name;
  • this value is used to construct the REST service URL: http://0.0.0.0:10101/database_name/collection
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_COLLECTION='events'
  • MongoDB collection name;
  • this value is used to construct the REST service URL: http://0.0.0.0:10101/database_name/collection
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_USERNAME=
  • MongoDB database username; if left blank no credentials will be used;
  • SPECS_MONITORING_EVENT_ARCHIVER_DATABASE_PASSWORD=
  • MongoDB database password; if left blank no credentials will be used;

We suggest to leave these parameters at defaults values.

REST Service Description

The REST interface is available at: http://0.0.0.0:10101/monitoring/events (using PUT, GET or DELETE). This is the default layout.

Check Service configuration section to learn how to customize the REST interface.

Insert into the database

Request call to insert a document (SPECS Monitoring format) into the database:

curl -4 -H Content-Type:application/json -X PUT -vv --data-binary @/dev/stdin localhost:10101/monitoring/events <<'EOS'
{
    "object": "nmap",
    "timestamp": 1434711558,
    "labels": [
        "userId-13",
        "jobId-5583f604545086115db13492"
    ],
    "component": "290b7267-a62b-42dd-9037-62b5e96fe719",
    "token": null,
    "type": "metric",
    "data": {
        "weakestCipher": "DES_CBC",
        "ecript2Level": 1
    }
}
EOS

Expected responses:

  • HTTP_201 - Document created
  • HTTP_500 - Request failed (in this case the output is in JSON format: {"code": "1", "message":"human readable message", "data": "JSON document returned by the MongoDB"}

Query the database

To query the archiver for data: make a GET request with filter attribute; filter attribute must be in JSON format (see MongoDB Find and MongoDB Query Operators for advanced usage);

curl -vv -X GET -G localhost:10101/monitoring/events -d 'filter={"object":{"$in":["nmap"]}}' -d'sort={"timestamp":-1}'

Filtering examples:

  1. Return all the events related to a component IDs: filter={"component" : {"$eq" : "uuid_of_the_component"}}
  2. Return all the events associated to a list of known component IDs: filter={"component" : {"$in" : ["uuid_of_the_component_1", "uuid_of_the_component_2", "uuid_of_the_component_N"]}}
  3. Return all the events of the same type: filter={"type" : {"$eq" : "type_name"}}
  4. Return all the events that have a known label (note: label field is a list): filter={"label" : {"$in" : ["label_1", "label_2", "label_N"]}}

Expected responses:

  • HTTP_200 - The query was successfull and JSON is returned; the JSON contains a list of SPECS Monitoring format that matches the query filter;
  • HTTP_500 - Query failed (in this case the output is in JSON format: {"code": "1", "message":"human readable message", "data": "JSON document returned by the MongoDB"})

Delete from the database

To delete events from the archiver: * make a DELETE request with the content the JSON format of the filter you want to apply for deletion:

curl -4 -H Content-Type:application/json -X DELETE -vv --data-binary @/dev/stdin localhost:10101/monitoring/events <<'EOS'
{
    "object": {
        "$in": [
            "nmap"
        ]
    }
}
EOS

Expected responses:

  • HTTP_200 - The query was successfull and JSON is returned; the JSON contains some explanations regarding the deletion of the requested range of events (JSON format: {"code": "1", "message":"human readable message", "data": "JSON document returned by the MongoDB"})
  • HTTP_500 - Query failed (in this case the output is in JSON format: {"code": "1", "message":"human readable message", "data": "JSON document returned by the MongoDB"})

Backup and restore

In directory backup/ are the backup scripts used for Event Archiver database migration. The scripts will take, by default, the information from etc/vars.sh for database connection.

Backup procedure

On the current Event Archiver instance execute the following command:

/opt/specs-monitoring-event-archiver/backup/export.sh /path/to/export/file

Restore procedure

On the target Event Archiver instance execute the following command, only after the /path/to/export/file has been copied locally:

/opt/specs-monitoring-event-archiver/backup/export.sh /path/to/import/file

If the database and collection are not present on the target instance the import scripts will automatically create them.

NOTICE

This product includes software developed at "Institute e-Austria, Timisoara", as part of the "SPECS - Secure Provisioning of Cloud Services based on SLA Management" research project (an EC FP7-ICT Grant, agreement 610795).

Developers:

Silviu Panica, silviu@solsys.ro / silviu.panica@e-uvt.ro

Copyright:

Copyright 2013-2015, Institute e-Austria, Timisoara, Romania
    http://www.ieat.ro/

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.