Output (gelf TCP) ever increasing delay

Graylog Central (peer support)
1

Hi,

We have a setup where we use a org.graylog2.outputs.GelfOutput to forward certain messages to an external script. After our last update (unfortunately a rather big leap, with a new server) we’re noticing that the delay between graylog receiving a message and that message is sent to the external script is increasing over time. When the external script is first connecting, the delay is close to 0, but after 1-2 hours there is a 15 minute delay. If nothing is done this delay seems to grow indefinitely. But if we do a reconnection from the script the delay drops to 0 again.

Environment:

  • OS: Debian 11, fully patched

  • Graylog Version: 4.3.7+05bccc7

Does anyone recognize the symptoms and/or have any hints on what might be the cause?

/Cheers
Torgny

2

Hello

I used the Output a couple times, but when I did there were no delays. Can I ask what script are you talking about?

3

Hi,

The script is a custom-made python script that listens on the TCP port specified in the output in graylog.
In that I record the timestamp of the message and the current time of receiving. This difference is what is slowly growing.
It has two parts, one that just receives the message and adds it to a queue that the second part is consuming from in another thread.

/Cheers
Torgny

4

Hello,

Understood, my apologies I don’t think Ill be any help on this issue. But if you believe this maybe a bug and have the data to support this try posting on GitHub.