Capture part of log starting from to as entire event

[kostyanius (Kostya)]

Hello,
Please advise is there any possibility to capture stream in graylog for future processing?
I mean starting “from” “to” using pattern and.

For example:

From: info:START_REQUEST
To: info: END_REQUEST

Currently graylog breaks into pieces entire event. Which should not be, but so as it is something like JSON (multiline) graylog consider each line of JSON lile a separate message.

Which function i should use this in order to avoid it and get each event intact from: info:START_REQUEST
To: info: END_REQUEST?

Thanks.

[megan201296 (Megan)]

What data are you shipping and via what method?

[kostyanius (Kostya)]

Hello @megan201296
It is data from docker container api.
I am using fluentd agent for shipping.
Here is my fluentd config file.

<source>
  type tail
  read_from_head true
  pos_file fluentd-docker.pos
  path /var/lib/docker/containers/*/*-json.log
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
</source>

<match docker.**>
  type copy
  buffer_type memory
  buffer_chunk_limit 256m
  buffer_queue_limit 128
  flush_interval 1s
  disable_retry_limit false
  retry_limit 17
  retry_wait 1s
  <store>
    type gelf
    host some.host.co #host name was changed
    protocol udp
    port 12201
  </store>
  <store>
    type stdout
  </store>
</match>

Data structure (see example in attachment):

[megan201296 (Megan)]

Ah, I apologize as I do not have experience with fluentd. Hopefully someone else will be able to assist.