Log order - docker, filebeat

Graylog Central (peer support)
, , ,
1

Hi,

I send logs from docker containers via filebeat to graylog Graylog 3.3.3+88d0659 on gl.example.com (Debian 11.0.8 on Linux 4.19.0-10-amd64). It works almost fine.

Docker logging driver is json-file

{"log":"2020-09-03 11:35:53 [urllib3.connectionpool] DEBUG: https://www.example.com HTTP/1.1\" 200 458\n","stream":"stderr","time":"2020-09-03T09:35:53.451685484Z"}

Filebeat config:

  filebeat.inputs:
    - type: container
      paths: 
        - '/var/lib/docker/containers/*/*.log'

    processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"

    - decode_json_fields:
        fields: ["message"]
        target: "json"
        overwrite_keys: true

    setup.template.settings:
      index.number_of_shards: 3
      
    logging.to_files: true
    logging.to_syslog: false

    output.logstash:
        hosts: ["gl.example.com:5044"]

Timestamp in graylog is truncated to 2020-09-03 11:35:53.451. How I can get full Timestamp from json file (2020-09-03 09:35:53.451685484)?

Thanks for any advices.
Pek

2

If I’m correct, graylog uses elastic search date data type, which can store date only in milisecond precision.
https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#date

3

So, is there way how to display logs in same order how is created on server?

4

If there is special a field with time of log created, yes you can replate timestamp with it, for example using pipeline rule.