Merge branch 'master' into 820-loki-protobuf

Author: romanchechyotkin

README.idoc.md

@@ -36,6 +36,8 @@ On MacBook Pro 2017 with two physical cores `file.d` can achieve the following t
 * 1.7GB/s in `files > devnull` case
 * 1.0GB/s in `files > json decode > devnull` case
 
+More benchmarks can be found [here](https://github.com/ozontech/file.d-bench).
+
 TBD: throughput on production servers.  
 
 ## Plugins
@@ -46,6 +48,9 @@ TBD: throughput on production servers.
 
 **Output**: @global-contents-table-plugin-output|links
 
+## Logging system
+
+We have also developed a scalable and high-performance database for logs — [seq-db](https://github.com/ozontech/seq-db). File.d can send data to [seq-db](https://github.com/ozontech/seq-db) using [Elasticsearch Bulk API](https://ozontech.github.io/seq-db-docs/#write-documents-to-seq-db). An example of sending data from file.d to [seq-db](https://github.com/ozontech/seq-db) can be found [here](/docs/examples.md#kafka-to-seq-db).
 
 ## What's next
 * [Quick start](/docs/quick-start.md)

README.md

@@ -36,6 +36,8 @@ On MacBook Pro 2017 with two physical cores `file.d` can achieve the following t
 * 1.7GB/s in `files > devnull` case
 * 1.0GB/s in `files > json decode > devnull` case
 
+More benchmarks can be found [here](https://github.com/ozontech/file.d-bench).
+
 TBD: throughput on production servers.  
 
 ## Plugins
@@ -46,6 +48,9 @@ TBD: throughput on production servers.
 
 **Output**: [clickhouse](plugin/output/clickhouse/README.md), [devnull](plugin/output/devnull/README.md), [elasticsearch](plugin/output/elasticsearch/README.md), [file](plugin/output/file/README.md), [gelf](plugin/output/gelf/README.md), [kafka](plugin/output/kafka/README.md), [loki](plugin/output/loki/README.md), [postgres](plugin/output/postgres/README.md), [s3](plugin/output/s3/README.md), [splunk](plugin/output/splunk/README.md), [stdout](plugin/output/stdout/README.md)
 
+## Logging system
+
+We have also developed a scalable and high-performance database for logs — [seq-db](https://github.com/ozontech/seq-db). File.d can send data to [seq-db](https://github.com/ozontech/seq-db) using [Elasticsearch Bulk API](https://ozontech.github.io/seq-db-docs/#write-documents-to-seq-db). An example of sending data from file.d to [seq-db](https://github.com/ozontech/seq-db) can be found [here](/docs/examples.md#kafka-to-seq-db).
 
 ## What's next
 * [Quick start](/docs/quick-start.md)

docs/examples.md

@@ -73,6 +73,40 @@ pipelines:
       default_short_message_value: "message isn't provided"
 ```
 
+## Kafka to seq-db
+The following config reads logs from kafka, processes them and sends into [seq-db](https://github.com/ozontech/seq-db) via Elasticsearch Bulk API.
+It assumes that logs are in docker json format.
+```yaml
+pipelines:
+  kafka_gelf_example:
+    input:
+      type: kafka
+      brokers: [kafka-broker-0.svc.local, kafka-broker-1.svc.local, kafka-broker-2.svc.local]
+      topics: [k8s-logs]
+
+    actions:
+    - type: json_decode                             # unpack "log" field 
+      field: log
+      metric_name: input
+      metric_labels: [k8s_label_app]                # expose input metrics to prometheus
+
+    # normalize                                     # unify log format
+    - type: rename
+      log: message
+      msg: message
+      ts: time
+      _ts: time
+      systemd.unit: service
+      syslog.identifier: service
+      k8s_label_app: service
+
+    output:
+      type: elasticsearch
+      endpoints:
+        - "http://seq-db-proxy:9002"
+```
+
+
 ## What's next?
 1. [Input](/plugin/input/README.md) plugins documentation
 2. [Action](/plugin/action/README.md) plugins documentation