Overview
Atlassian Sourcetree is a free Git and Mercurial client for Windows.
Atlassian Sourcetree is a free Git and Mercurial client for Mac.
Fluent::Plugin::Mongokpi
Description
Fluent BufferedOutput plugin: counting chunk, inserting counts to MongoDB version 2.4 or newer MongoDB is required (using setOnInsert command)
Installation
$ gem install fluent-plugin-mongokpi
Configuration
config | default | info |
---|---|---|
address | localhost:27017 | address of mongodb (for replicaset, plz set like localhost:27017,localhost:27018,localhost:27019) |
db | kpidb | db name of mongodb |
collection | kpiColyyyymmdd | collection name of mongodb (yyyymmdd will be replaced based on time) |
write_concern | mongodb connection option | |
name | mongodb connection option | |
read | mongodb connection option | |
refresh_mode | mongodb connection option | |
refresh_interval | mongodb connection option | |
kpi_type | use 'access' for activating access count | |
kpi_unit | min | time unit of count [min,hour,day] |
time_key | this time_key field or time of record is used | |
time_format | (with time_key) the value of time_key will be parsed with this format | |
count_key | used like group by (for multiple keys, plz set like site,host,instance) | |
count_name | count | the count result will be set to this field |
response_threshold | 1000000 | (with kpi_type = access) used as threshold of response time |
f_code | code | (with kpi_type = access) field name of response code in the log records |
f_response_time | response_time | (with kpi_type = access) field name of response time in the log records |
MongoDB document
field | notification | info |
---|---|---|
_id | time + value of ${count_key} | |
yyyymmdd | ||
hh | (with kpi_unit = min or hour) | |
mm | (with kpi_unit = min) | |
${count_key} | ||
${count_name} | count | |
count1xx | (with kpi_type = access) count of ${f_code} = 1XX | |
count2xx | (with kpi_type = access) count of ${f_code} = 2XX | |
count3xx | (with kpi_type = access) count of ${f_code} = 3XX | |
count4xx | (with kpi_type = access) count of ${f_code} = 4XX | |
count5xx | (with kpi_type = access) count of ${f_code} = 5XX | |
responseTimeAve | m(_ _)m |
(with kpi_type = access) average of ${f_response_time} |
responseTimeMax | m(_ _)m |
(with kpi_type = access) maximum of ${f_response_time} |
responseTimeMin | m(_ _)m |
(with kpi_type = access) minimum of ${f_response_time} |
responseTimeSum | (with kpi_type = access) sum of ${f_response_time} | |
countOver | (with kpi_type = access) count of the access: ${f_response_time} >= ${response_threshold} | |
okRatio | m(_ _)m |
(with kpi_type = access) (count - countOver) / count |
counter[60] | (with kpi_type = access & kpy_unit = min) qps for each second | |
qpsAve | m(_ _)m |
(with kpi_type = access & kpy_unit = min) average of qps |
qpsMax | m(_ _)m |
(with kpi_type = access & kpy_unit = min) maximum of qps |
qpsMin | m(_ _)m |
(with kpi_type = access & kpy_unit = min) minimum of qps |
m(_ _)m
: these parameters are updated by non-atmic queries, then not reliable in multi processing
Samples
sample: access log (simple)
config
<match ***> type mongokpi kpi_type access kpi_unit hourly count_key site time_key request_time time_format %d/%b/%Y:%H:%M:%S %Z </match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find() { "_id" : "git2013112520", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0, "hh" : 20, "okRatio" : 1, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 } { "_id" : "blog2013112520", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0, "hh" : 20, "okRatio" : 1, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 } { "_id" : "total2013112520", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0, "hh" : 20, "okRatio" : 1, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: access log (detail)
config
<match ***> type mongokpi kpi_type access count_key site time_key request_time time_format %d/%b/%Y:%H:%M:%S %Z </match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"} {"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find() { "_id" : "git201311252003", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0, "counter" : [ 0,0,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 } { "_id" : "blog201311252003", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0, "counter" : [ 0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 } { "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0, "counter" : [ 0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 4, "qpsMin" : 0, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: app log
config
<match ***> type mongokpi count_key site </match>
log
{"gerogero":"Let's get Rocking!","site":"git"} {"gerogero":"Let's get Rocking!","site":"git"} {"gerogero":"Let's get Rocking!","site":"git"} {"gerogero":"Let's get Rocking!","site":"blog"} {"gerogero":"Let's get Rocking!","site":"blog"}
MongoDB
db.kpiCol20131125.find() { "_id" : "git201311252248", "count" : 3, "hh" : 22, "mm" : 48, "site" : "git", "yyyymmdd" : 20131125 } { "_id" : "blog201311252248", "count" : 2, "hh" : 22, "mm" : 48, "site" : "blog", "yyyymmdd" : 20131125 } { "_id" : "total201311252248","count" : 5, "hh" : 22, "mm" : 48, "site" : "total", "yyyymmdd" : 20131125 }
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request