Overview
Atlassian Sourcetree is a free Git and Mercurial client for Windows.
Atlassian Sourcetree is a free Git and Mercurial client for Mac.
Fluent::Plugin::Mongokpi
Description
Fluent BufferedOutput plugin: counting chunk, inserting counts to MongoDB version 2.4 or newer MongoDB is required (using setOnInsert command)
Installation
$ gem install fluent-plugin-mongokpi
Configuration
| config | default | info |
|---|---|---|
| address | localhost:27017 | address of mongodb (for replicaset, plz set like localhost:27017,localhost:27018,localhost:27019) |
| db | kpidb | db name of mongodb |
| collection | kpiColyyyymmdd | collection name of mongodb (yyyymmdd will be replaced based on time) |
| write_concern | mongodb connection option | |
| name | mongodb connection option | |
| read | mongodb connection option | |
| refresh_mode | mongodb connection option | |
| refresh_interval | mongodb connection option | |
| kpi_type | use 'access' for activating access count | |
| kpi_unit | min | time unit of count [min,hour,day] |
| time_key | this time_key field or time of record is used | |
| time_format | (with time_key) the value of time_key will be parsed with this format | |
| count_key | used like group by (for multiple keys, plz set like site,host,instance) | |
| count_name | count | the count result will be set to this field |
| response_threshold | 1000000 | (with kpi_type = access) used as threshold of response time |
| f_code | code | (with kpi_type = access) field name of response code in the log records |
| f_response_time | response_time | (with kpi_type = access) field name of response time in the log records |
MongoDB document
| field | notification | info |
|---|---|---|
| _id | time + value of ${count_key} | |
| yyyymmdd | ||
| hh | (with kpi_unit = min or hour) | |
| mm | (with kpi_unit = min) | |
| ${count_key} | ||
| ${count_name} | count | |
| count1xx | (with kpi_type = access) count of ${f_code} = 1XX | |
| count2xx | (with kpi_type = access) count of ${f_code} = 2XX | |
| count3xx | (with kpi_type = access) count of ${f_code} = 3XX | |
| count4xx | (with kpi_type = access) count of ${f_code} = 4XX | |
| count5xx | (with kpi_type = access) count of ${f_code} = 5XX | |
| responseTimeAve | m(_ _)m |
(with kpi_type = access) average of ${f_response_time} |
| responseTimeMax | m(_ _)m |
(with kpi_type = access) maximum of ${f_response_time} |
| responseTimeMin | m(_ _)m |
(with kpi_type = access) minimum of ${f_response_time} |
| responseTimeSum | (with kpi_type = access) sum of ${f_response_time} | |
| countOver | (with kpi_type = access) count of the access: ${f_response_time} >= ${response_threshold} | |
| okRatio | m(_ _)m |
(with kpi_type = access) (count - countOver) / count |
| counter[60] | (with kpi_type = access & kpy_unit = min) qps for each second | |
| qpsAve | m(_ _)m |
(with kpi_type = access & kpy_unit = min) average of qps |
| qpsMax | m(_ _)m |
(with kpi_type = access & kpy_unit = min) maximum of qps |
| qpsMin | m(_ _)m |
(with kpi_type = access & kpy_unit = min) minimum of qps |
m(_ _)m : these parameters are updated by non-atmic queries, then not reliable in multi processing
Samples
sample: access log (simple)
config
<match ***> type mongokpi kpi_type access kpi_unit hourly count_key site time_key request_time time_format %d/%b/%Y:%H:%M:%S %Z </match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git2013112520", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog2013112520", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total2013112520", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: access log (detail)
config
<match ***> type mongokpi kpi_type access count_key site time_key request_time time_format %d/%b/%Y:%H:%M:%S %Z </match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git201311252003", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"counter" : [ 0,0,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252003", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
"counter" : [ 0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"counter" : [ 0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 4, "qpsMin" : 0, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: app log
config
<match ***> type mongokpi count_key site </match>
log
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"blog"}
{"gerogero":"Let's get Rocking!","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git201311252248", "count" : 3, "hh" : 22, "mm" : 48, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252248", "count" : 2, "hh" : 22, "mm" : 48, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252248","count" : 5, "hh" : 22, "mm" : 48, "site" : "total", "yyyymmdd" : 20131125 }
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request