GELF message ERROR in server.log

Graylog Tech Challenges
1

Hi team,

I’m new to graylog and I’ve set up inputs from servers to graylog server via fluentd UDP using GELF.
I have fine messages in Graylog however server log is full of ERROR messages like:

ERROR [EnvelopeMessageAggregationHandler] Caught exception while decoding type of GELF packet: GELF message is too short. Not even the type header would fit.

Input settings GELF UDP

bind_address: 0.0.0.0
decompress_size_limit: 8388608
number_worker_threads: 2
override_source: <empty>
port: 12202
recv_buffer_size: 262144

Fluentd config

<match graylog2.**>
  @type gelf
  host x.x.x.x
  port 12202
  <buffer>
    flush_interval 5s
  </buffer>
</match>

  <source>
    @type tail
    path /.../*.log
    tag graylog2
    read_from_head true
  <parse>
    @type none
    time_format %d/%m/Y %H:%M:%S,%L%z
  </parse>
  </source>
  <match sample.**>
    @type forward
    heartbeat_type udp
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s
    <server>
      name X.server
      host x.x.x.x
      port 12202
      weight 60
    </server>
  </match>

I’m using latest graylog version 4.1

I tried to debug it many ways nevertheless there is no clue which could provide me more info to solve this issue. Please help, many thanks.

2

Hello && Welcome

Are you getting messages in the GELF /UDP input?
Can you try using TCP to connect to Graylog?

UDP is stateless (fire and forget) so there isn’t any error checking to see if a GELF packet was received or not.

3

Hi,

I’m getting messages in the GELF /UDP input, there is no problem.
I can try TCP and let you know.

I know that about UDP, but it shouldn’t generate those errors in server.log em I right?

4

Hello,
Just an Idea when something like this happens in my environment, I will test out other Inputs like Raw/Plaintext UDP input, etc…

Sometime it does not, with a three way handshake you can find out connection/error issue on both sides (sender & receiver).

I understand, so no bad problems your just seeing these errors in the Logs.
I found something similar

It might be the client sending messages to your GELF input. It’s hard to tell without testing.

1 Like
5

Hi gsmith,

I tried the TCP connection without TLS to Graylog, which is working on the fluentD side however the graylog input is probably setup wrong (I made new input GELF TCP - left it by default just choose the correct port for TCP connection). It didn’t recieve any message (I’ve checked by TCP dump on graylog server the communication and it recieves messages from the client server)

I’ve checked one UDP connection/input and it works fine and it isn’t missing any data - I assume the rest of UDP connections works fine too since they all have the very same configuration - just ports are different.

In additon I’ve been reading the other topics which you provided without solution, which was the reason why I decided to post new topic.

If is necessary to post some config, test or output, there is no problem.

Thanks in advance.

EDIT: Well I checked the server.log today and it has been solved by itself over the weekend.
There are no ERRORs messages now.
Many thanks for effort gsmith, I appreciate it!

J.

2 Likes
6

Glad I could help.

Sometimes you just have to wait, it probably rotated you index, just an idea.

2 Likes
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.