Daily Challenge: Fail to transfer only ERROR and WARN log using fluentbit Loglevel

[dscryber (David Sciuto)]

Can you help this member answer a question?

[deepakkumardubey (DK)]

Hello Friends,

I am using Fluentbit to transfer a log from a file to another destination.

As per Fluentbit documentation, we have used the Log level setting in the service section of fluentbit.conf. it looks like below.

[SERVICE]
Flush 1
Daemon Off
Log_Level warning
Parsers_File /etc/td-agent-bit/parsers.conf

As I expect only ERROR and WARN logs to be transmitted to the destination.
But it’s picking up all other Loglevel like INFO, DEBUG etc.

Please suggest to me why and how can I fix it.

[dscryber (David Sciuto)]

Hello, DK,

I noticed your post wasn’t answered. I want to help by moving it to our Daily Challenges where we can expect to get more expert community member eyes on it. If you’re able, please send more information on the process you’ve used and specifically where Graylog was involved.

I’m not sure if you’ve checked out this document, but let me know if it helps:

GELF

You’ll note that at the bottom of the documentation page, it describes what you should see:

Now, this is what happens to this log:

1. Fluent Bit GELF plugin adds "version": "1.1" to it.

2. The Nest Filter, unnests fields inside log key. In our example, it puts data alongside stream and time .

3. We used this data key as Gelf_Short_Message_Key ; so GELF plugin changes it to short_message .

4. ​Kubernetes Filter adds host name.

5. Timestamp is generated.

6. Any custom field (not present in GELF Payload Specification) is prefixed by an underline.

Finally, this is what our Graylog server input sees:

{“version”:“1.1”, “short_message”:“This is an example.”, “host”: “”, “_stream”:“stderr”, “timestamp”:1565770310.000199}