Hello everyone,
I’m using Graylog Open and currently have some Syslog inputs, each mapped to a corresponding stream. I’ve just added three GELF outputs over TCP. I’ve noticed that if the target endpoint of a GELF output crashes, all logs in Graylog get stuck in the journal. Normal writing only resumes after the GELF output is restored.
Is this expected behavior? Also, is it required that all GELF outputs remain reachable to ensure logs can be written normally?
Thank you.
I did a bit of a quick research, but I am using Enterprise so I could not test it.
Yes, expected behaviour, a design limitation, not a misconfiguration.
Graylog serves all outputs from one shared thread pool, and the GELF output has no buffer of its own. When the far end stops accepting, the send call blocks and holds its thread. All output threads park within seconds, the journal stops draining. Graylog staff confirmed it: the GELF and STDOUT outputs have no buffers, the buffered ones are Enterprise. Graylog Community
Because the pool is global, it stalls indexing for every stream, not just the affected one. And once the journal hits its size or age limit, old segments are droppe, real data loss.
So yes: with GELF TCP on Open, all three targets need to stay reachable. No per-output timeout or drop-on-failure exists.
Possible Fixes: run a local relay (Vector, Fluent Bit, syslog-ng) on the Graylog host, point the outputs at 127.0.0.1, and let it handle buffering and retries. Or use GELF UDP, lossy, but can’t backpressure. Alert on journal utilization either way.
Thank you Austin! I am using udp instead tcp now…
