Graylog stops processing all incoming trafic all the sudden

[H2Cyber]

My setup consists of a GL VM and an EL VM, both running on Ubuntu 18.04. All inputs are setup as Syslog UDP listeners. Receive Buffer Size has been increased both at the OS (/etc/sysctl.conf) and Graylog level (input configuration).

I am constantly running into an annoying situation wherein Graylog stops procssing all incoming logs, seemingly after receiving a spike of UDP messages. Graylog won’t process any new message unless I intervene.

I have been troobleshooting the issue for a few days and was able to reproduce it by hammering Graylog with UDP packets (using logger on a ubuntu VM). When input processing stops, I have tcpdump results which prove that UDP packets are reaching the Graylog VM. I have checked the GL and ES logs for any errors, and found none. When the issue happens, Input, Process and Output buffers all drop to 0%. Journal utilisation is at 2% max.

Interrestingly, restarting any of my inputs resolves the issue for all inputs. i.e. if I restart my input UDP/1514 (and only this one); all my inputs (UDP/1514, UDP/2514, UDP/3514, etc.) resume working normally.

Did anybody encounter a similar issue ?

[jan (Jan Doberstein)]

he @H2Cyber

from your description it could be anything. something on OS level, or in the components of Graylog. The field is rich and broad that need to be watched. Firewall? Kernel? Graylog? Elasticsearch? You need to check every component when the system is not responding what exactly is not working. Does Graylog receive the message or is this stuck before? Is the message processed in Graylog but not delivered to Elasticsearch?

Sorry I can’t point to one item to solve the problem.