Filebeat without sidecar


(rm) #1

hi,

i have been asking around everywhere, and it seems i fell victim to a very basic misunderstanding somehwhere, and i can’t seem to be able to figure out where on my own.

i have a rancher cluster, with several environments and with heaps of services running, and i want to collect log data from those. also, this is running on top of EC2, so i built a small filebeat container like this:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition:
            not.contains:
              docker.container.labels.log_enabled: "false"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              json.message_key: log  
              json.keys_under_root: true
              json.add_error_key: false
processors:
 - add_cloud_metadata: ~
 - add_docker_metadata: ~ 
output.logstash: 
    hosts: ["${LOGSTASH_HOST}:${LOGSTASH_PORT}"]
    bulk_max_size: 1024

i changed the logstash output to a file output temporarily in between, and it writes nice json lines, with all the metadata i want.

in graylog, i create a Beats input, and configured the variables above to point to that input. using tcpdump, i verified that filebeat is sending more or less identical json objects to this input.

i have not other configuration (yet). so i can see the incoming messages in “All Messages” when logged as admin. the thing is that those message do not have any metadata in them. all the extra fields are gone.

i am using:
docker.elastic.co/filebeat/filebeat:6.5.1 (also tried 5.6.13)

and
graylog/graylog:2.4.6-1

i also looked at the source code for the beats plugin, and it very much looks like

BeatsCodec.addFlattened()

should just add all fields from the JSON object as dot-separated flattened fields to the message. but nothing, nada, zilch, 没有. i am hitting a wall here, and would be extremely grateful if anyone could point out to me what i’m not getting.

regards

ruben


(Jan Doberstein) #2

What I did not understand - you talked about filebeat in the beginning and later that you are using logstash and Graylog.

You can, without a Problem use filebeat ingest into Graylog without any logstash between them. Did you verify that your filebeat can reach Graylog and the input port?


(rm) #3

hi jan,

thank you for your reply. so, the setup is as follows (very simple):

  • filebeat (autodiscover from docker daemon)
  • logstash output configured in filebeat
  • beats input configured on graylog

i can see full json objects with all metadata fields arriving on the given port (in my case 8044) using tcpdump. i can also see the messages in graylog, but they only have the bare minimum of fields set (i.e. only “facility”, “timestamp”, “source” and “message” are set)

the docker container name or image name, the compute host it was running on and all the other metadata is not set (although they ARE present in the json object that filebeat sends to graylog).

.rm


(rm) #4

oh, also i noticed that i made a mistake - i am using the filebeat docker image to collect the logs, not logstash. sorry, i copied the wrong image name there.


(Jan Doberstein) #5

got it now - we have a new beats input that should have no issues with the new meta fields added. But I’m not 100% sure on that - if that is really the fix for your issue or if this is a new.

Did you have the option to test with a 3.0 alpha release if the new beats input solve your problem?


(rm) #6

hi,

i can see this commit:

https://github.com/Graylog2/graylog-plugin-beats/commit/1b3c51f2446e406eac1adc37be0ad5c122674e19

which is where the

BeatsCodec.addFlattened()

method i was referring to got added (it seems to use MapUtils.flatten prior to that).

that commit is already rather old, but it seems this hasn’t been merged into 2.4 or 2.5? so … how would i add this to 2.4? i tried the alpha of 3 briefly, but had some minor issues and then reverted to 2.4 … i can try again, though. is there a timeline for the release of 3.x? i don’t really want to live with an alpha version for the next five years … :blush:

also, i think this:

https://github.com/Graylog2/graylog2-server/issues/5313

is related.

.rm


(Jan Doberstein) #7

the new Beats Input is part of 3.0 that will be releases on 14.2.2019


(rm) #8

hi,

yes, i can confirm that this seems to solve it. with the alpha and a new input configured, everything works exactly as expected (see robert’s github issue linked above). i now have to decide how long i can live with running an alpha release …

thank you!

.rm


(system) closed #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.