Hi folks, After some hilarity with processbuffers getting stuck and no longer outputting, I’m now looking at 3 graylog nodes with 300 million messages in the journal each, with no output to ES. All process buffers are empty/idle (according to processbuffer dump). A restart of the graylog server do…

From what I have been reading, the NullPointerException leads me to believe the journal is corrupt but I can’t find anything that says that solidly. Its a stretch but you could stop the server, move the journal data out, start Graylog, verify its running smooth then turn it off and swap the old dat…

I have indeed moved the journal out of the way and restarted the Graylog instances, and they started okay, then got bogged down again where the process buffers stay full (an older, other issue I’ve had), but it’s sort of fixed itself. It gets stuck occasionally, seems some apps send it one message t…

This reminded me of a similar issue I had a while back where I Graylog would lock up and I could see things listed in a process buffer dump (“I see dead things”) the post was here - hopefully that will have some info that will help (not with corrupt journal tho…) In short I believe it was a GROK w…

We don’t really use grok patterns anymore, I’ve managed to get our developers to log everything in JSON so all we do is extract it (which, well, I’ve got issues with some of that too but that’s another story altogether) and run pipelines to massage it a bit more.

Hello Service JournalReader [FAILED] has failed in the RUNNING state.java.lang.NullPointerException: null Not sure if this will help, but I have seen this resolved by upgrading JAVA and/or increasing JVM heap. But most of the time its some type of GROK/REGEX not correct.

JVM heap is already at 16Gb, not sure increasing it further will really help, but… I’ll give it a shot!

Based on the post here Disk journal stopped working after disk filled opened 02:42PM - 04 Aug 16 UTC (UTC) [JulioQc] JulioQc improvement …

Another fun problem: all processbuffers remain empty

Graylog Central (peer support)

dscryber (David Sciuto) August 29, 2022, 4:16pm 3

Yes. Thanks, @tmacgbay

And again, Graylog does not output anything

Topic		Replies	Views
Problems in graylog about Disk Journal and Process buffer Graylog Central (peer support)	9	3564	February 17, 2021
Graylog nodes stop outputting/fill up buffers Graylog Central (peer support)	14	6379	April 22, 2020
Graylog server with full journal Graylog Central (peer support)	0	699	September 24, 2020
Process buffer 100% and nothing helps until restart Graylog Central (peer support)	1	1846	January 16, 2019
Messages stop being read from disk journal Graylog Central (peer support)	12	1003	June 28, 2022

Another fun problem: all processbuffers remain empty

Related topics