If you wanted to use Filebeat with add_docker_metadata processor, rejoice, there's a dirty solution to make it work

[benvanstaveren]

See the subject. In our infrastructure we have a need to grab docker logs by way of Filebeat (due to gelf driver in Docker not doing TLS, and Logstash being a… well. No comment, but it isn’t pretty). One thing we require is that we know what type of app is running in the container, due to a few shenanigans by our developers we can’t rely on the image name (because re-tagging a new release with an already used tag screws it up greatly), and we need the content of a few labels as well.

So, anyone that’s tried pointing Filebeat with the add_docker_metadata processor at Graylog beats input knows that any fields that are not “message” and “fields” are quietly dropped. It seems that that will get fixed in 3.0 but that’s a few months out, and probably a few more months before we upgrade our relatively new setup.

Long story short, here’s how you do it:

processors:
  - drop_fields:
      fields: [ 'message' ]
  - rename:
      fields:
        - from: "json.log"
          to: "message"
      ignore_missing: true
      fail_on_error: false

    filebeat.autodiscover:
      providers:
        - type: docker
          templates:
            - condition:
                regexp:
                  docker.container.labels.supercoolapp: "^.+$"
              config:
                - type: log
                  json.message_key: "log"
                  fields:
                    docker_container_image: "${data.docker.container.image}"
                    docker_container_id: "${data.docker.container.id}"
                    docker_container_name: "${data.docker.container.name}"
                    docker_container_labels_supercoolapp: "${data.docker.container.labels.supercoolapp}"
                    server_type: supercoolappserver
                    type: theappcontainer
                  paths:
                    - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                    - "/var/lib/docker/*/containers/${data.docker.container.id}/*.log"
                  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
                  multiline.negate: false
                  multiline.match: after
            - condition.not:
                regexp:
                  docker.container.labels.supercoolapp: "^.+$"
              config:
                - type: log
                  json.message_key: "log"
                  fields:
                    docker_container_image: "${data.docker.container.image}"
                    docker_container_id: "${data.docker.container.id}"
                    docker_container_name: "${data.docker.container.name}"
                    server_type: supercoolappserver
                    type: other
                  paths:
                    - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
                    - "/var/lib/docker/*/containers/${data.docker.container.id}/*.log"

And now we have part of the docker metadata coming in to Graylog, parsed properly by the Beats input, and our team rejoiced and was very happy that we could finally use Graylog for all the things. It works with the filebeat shipped with the collector sidecar as well.

Figured this may come in handy at some point for someone, somewhere.