GELF input from Fluent-bit

Graylog Central (peer support)
1

1. Describe your incident:
Regular output of error messages in the container log for Graylog, in the format of -
WARN : org.graylog2.inputs.codecs.GelfCodec - GELF message <b98ed090-a868-11df-a553e-02a2ac1265405> (received from <1.2.3.4:24996>) is missing mandatory "host" field.

These appear every few seconds from every host that had Fluent-bit deployed in a k3s cluster.
The log entry itself, shows correctly in the Graylog GUI.

2. Describe your environment:

  • OS Information: Debian/Docker

  • Package Version: Graylog v5.2

  • Service logs, configurations, and environment variables:

  output-graylog.conf: |
    [OUTPUT]
        Name                    gelf
        Match                   *
        Host                    syslog.domain.com
        Port                    12204
        Gelf_Short_Message_Key  log
        Gelf_Host_Key           host
        Mode                    tcp

3. What steps have you already taken to try and solve the problem?
Multiple configuration changes to the fluent-bit config maps.
Unless something is badly documented or a bug, everything appears to be how its meant to be configured.

Attempted input overrides in Graylog too, but to no avail.

4. How can the community help?
Help identify root cause or workaround to remove log noise from graylogs container log output.

2

Some additional reading, GELF - Fluent Bit: Official Manual indiciates -
If you're using Fluent Bit in Kubernetes and you're using [Kubernetes Filter Plugin](/manual/pipeline/filters/kubernetes), this plugin adds `host` value to your log by default, and you don't need to add it by your own.
For my config, i am indeed using the filter plugin -

  input-kubernetes.conf: |
    [INPUT]
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Parser             docker
        DB                 /var/log/flb_graylog.db
        DB.Sync            Normal
        Docker_Mode        On
        Buffer_Chunk_Size  512KB
        Buffer_Max_Size    5M
        Rotate_Wait        30
        Mem_Buf_Limit      30MB
        Skip_Long_Lines    On
        Refresh_Interval   10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Key       log
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off		
        Annotations         Off
        Labels              On

So based on that note in the docs, this shouldnt be an issue at all?

3

IMO its a bug as many people seem to report the same issue. I’m unfortunately not familiar enough with both kubernetes and fluentbit to understand how to troubleshoot it though.

What version of fluentbit are you using?

4

Hi,
Latest release of Fluent-Bit, 2.2.1.
From what i can see in the repos for Fluent-Bit, it appears to be a reasonably common issue (#1, #2), but without a solution from their side.
 
To attack this a different way, is there a way to suppress these messages in GrayLog’s STDOUT?
Setting the log level to ‘Error’ in the GUI appears to work, but it doesn’t stick after each restart of the container unfortunatelly.
 
Thanks!

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.