I have a single combined Graylog 3.0.0-12 / Elasticsearch 6.6.0 single instance (4vcpu/8GB ram) (Virtualized on Proxmox) where at 10am EST/1500UTC every day Graylog stops processing messages with no error message.
Both Graylog and Elasticsearch have been given 2GB of heap, monitoring shows no memory or heap exhaustion.
At 10am (EST) a single Graylog thread will start spinning at 100% cpu and the process buffer will start filling up, some messages are processed but once the buffer fills up no new messages are written to Elasticsearch. Also once the process buffer is filled up a second Graylog thread starts spinning at 100% cpu.
No log entries (at Debug level) are in Graylog’s server.log, nothing to note in Elasticsearch logs as well.
Data is a few servers’s audit logs in one injest with some Grok matching rules.
Restarting Graylog will allow the messages to be processed (with no apparent dropped messages).
Any insight into further troubleshooting steps or resolving this issue?