Bitbucket is a code hosting site with unlimited public and private repositories. We're also free for small teams!

Close

syslog2couchdb

syslog2couchdb is a tool that converts syslog messages into JSON documents and stores them in a CouchDB database.

syslog2couchdb inserts JSON documents in bulk mode to keep best performance.
To achieve that it stores some messages in memory before it connects to database.
Also, a timer periodically trigger an insertion to avoid keeping data in memory for too long time.

ChangeLog

  • Version 0.4.0
    • Listener: Add support for more than one listener.
    • Listener: Support for protocols other than UDP: TCP, UNIX, ...
    • Pusher: Support for adding a static JSON key in document (see --static-json-key and --static-json-value).

Big Picture

Big Picture

Listener options (1)

  • --listen "proto:ip:port"[, "proto:ip:port" [,...] ] (V0.3.x support only UDP and one listener)
    Bind listener(s) to desired address(es). Default is:

    --listen="udp:127.0.0.1:5140"
    
  • --max-message-size size_in_bytes
    Set the syslog packet input buffer size in bytes. If the message is larger then it will be truncated. For UDP transport, maximum is ~64k. Default is:

    --max-message-size=32768
    

Messages Queue options (2)

  • --messages-queue-length max_messages_in_queue
    Set the maximum number of messages in queue. Default is:

    --messages-queue-length=1024
    

Worker Pusher options (3)

  • --format syslog_format_name
    Select a predefined syslog format decoder. Default is:

    --format="RsyslogForward"
    
    • Available formats (case sensitive) : (syslog_format_name : regular expression):
      • RsyslogForward : ^<(?P<Pri>\d+)>(?P<TimestampRFC3339>\S+)\s+(?P<Remote>\S+)\s+(?P<Progname>\S+)\s+(?P<Message>.*)$
  • --custom-format regular_expression
    Set a custom regular expression to decode syslog message. Overrides --format option. Default is:

    --custom-format=""
    
    • Available named (case sensitive) capturing group for decoder are:
      • Version : Syslog version. Default = 0 (integer)
      • Pri : Syslog Priority. Facility and Severity are produced from Pri. Default = 0 (integer)
      • TimestampRFC3339 : Syslog timestamp in RFC3339 format. TimestampRFC3339 fills the following JSON document keys: DateStr, Year, Month, Day, Hour, Min, Sec, TZName, TZOffset, TS, TSnano.
      • TimestampRFC1123 : Syslog timestamp in RFC1123 format. TimestampRFC1123 fills the following JSON document keys: DateStr, Year, Month, Day, Hour, Min, Sec, TZName, TZOffset, TS, TSnano.
      • TimestampRFC850 : Syslog timestamp in RFC850 format. TimestampRFC850 fills the following JSON document keys: DateStr, Year, Month, Day, Hour, Min, Sec, TZName, TZOffset, TS, TSnano.
      • TimestampRFC822 : Syslog timestamp in RFC822 format. TimestampRFC822 fills the following JSON document keys: DateStr, Year, Month, Day, Hour, Min, Sec, TZName, TZOffset, TS, TSnano.
      • DateStr : Raw (undecoded) timestamp. Preferably use TimestampRFCxxx. Default = "" (string)
      • Year : Default = 0 (integer) Year of message date
      • Month : Default = 0 (integer) Month of message date
      • Day : Default = 0 (integer) Day of message date
      • Hour : Default = 0 (integer) Hour of message date
      • Min : Default = 0 (integer) Minute of message date
      • Sec : Default = 0 (float) Second of message date
      • TZName : Default = "" (string)
      • TZOffset : Default = 0 (integer)
      • TS: Default = 0 (integer) Number of seconds elapsed since January 1, 1970 UTC
      • TSnano: Default = 0 (integer) Number of nanoseconds elapsed since January 1, 1970 UTC (OSes dependent).
      • Remote : Default = "" (string) Hostname or IP of sender
      • Progname : Default = "" (string) Process Name
      • Pid : Default = 0 (integer) Process ID
      • Message : Default = "" (string) The message
  • --workers number_of_workers
    Set the number of WorkerPusher. Default is:

    --workers=2
    
  • --trigger-insert-messages number_of_messages
    Set the number of messages to keep in memory before a bulk insertion. Default is:

    --trigger-insert-messages=10
    
  • --trigger-insert-period time_in_milliseconds
    Time in milliseconds to wait between each periodic bulk insertion. Min = 1 ms. Max = 1 Minute. Default is 10s:

    --trigger-insert-period=10000
    
  • --couchdb-url url_to_database
    Set the URL of the database to store syslog documents. Default is:

    --couchdb-url="http://localhost:5984/syslog"
    
  • --static-json-key "key" (V0.3.x doesn't support static-json-key.)

    --static-json-key="doctype"
    
  • --static-json-value "value" (V0.3.x doesn't support static-json-value.)
    NB: JSON string MUST BE surrounded by double quotes.

    --static-json-value='["syslog2couchdb","apache-log"]'
    

Global option

  • --debug level

    --debug=0
    
       1 : Debug Startup
       2 : Debug Listener
       4 : Debug Queue
       8 : Debug Worker
    + 16 : Debug Pusher
    ---------------------
    = 31 : Debug ALL
    

Example:

Assuming:

  1. A CouchDB database "test1" is available on localhost:5984.
  2. Your local rsyslog has the following configuration added (send syslog message from program logger to UDP@localhost:5140 in RSyslogForward format):

    if $programname == 'logger' then @127.0.0.1:5140;RSYSLOG_ForwardFormat
    if $programname == 'logger' then ~
    
  3. syslog2couchdb is running

    ./syslog2couchdb --listen="udp:127.0.0.1:5140" --couchdb-url="http://localhost:5984/test1" --static-json-key=doctype --static-json-value='["syslog2couchdb","apache-log"]'
    
  4. And you produce a syslog message with logger:

    echo "A syslog message" | logger
    

Then:

  1. syslog2couchdb will receive this message:

    <13>2012-06-23T16:46:04.557553+02:00 lutetium logger: A syslog message
    
  2. and push a new document in CouchDB database "test1":

    {
       "_id": "e1a5053ceb88275a1eecd57b5b005d26",
       "_rev": "1-9cc67196b1b0236c19c9cf57c310bff6",
       "Version": 0,
       "Pri": 13,
       "Severity": 5,
       "Facility": 1,
       "DateStr": "2012-06-23 16:46:04.557553 +0200 CEST",
       "TS": 1340462764,
       "TSnano": 1340462764557553000,
       "Year": 2012,
       "Month": 6,
       "Day": 23,
       "Hour": 16,
       "Min": 46,
       "Sec": 4.557553,
       "TZName": "CEST",
       "TZOffset": 7200,
       "Remote": "lutetium",
       "Pid": 0,
       "Progname": "logger:",
       "Message": "A syslog message",
       "doctype": [
           "syslog2couchdb",
           "apache-log"
       ]
    }
    

Build from sources

If not already installed, fetch and install Go1 compiler.

Download syslog2couchdb.
Extract archive and cd to sources directory. Then type: go build -ldflags -s

Licence

Copyright (C) 2012 Stephane Bunel. All rights reserved.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

Recent activity

Stéphane Bunel

Commits by Stéphane Bunel were pushed to StephaneBunel/syslog2couchdb

62c11af - Version 0.4.0 Listener: Add support for more than one listener. Listener: Support for protocols other than UDP: TCP, UNIX, ... Pusher: Support for adding a ...
Stéphane Bunel

Commits by Stéphane Bunel were pushed to StephaneBunel/syslog2couchdb

fe3ff6d - ADD new attributes in JSON documents: TS and TSnano TS is number of seconds elapsed since January 1, 1970 UTC TSnano number of nanoseconds elapsed ...
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.