My Graylog setup 3 VMs 12 core 32 GB memory Applications - Zookeeper, Kafka, MongoDB, Graylog Zookeeper JVM settings - -Xmx512M -Xms512M Kafka JVM settings - -Xmx1G -Xms1G Mongo JVM settings - Default Graylog - -Xms8g -Xmx8g processbuffer_processors = 12 outputbuffer_processors = 1 ou…

Perhaps related to this post where it was a GROK issue? the post at the end shows the beginnings of how it was tracked down… [image] Graylog nodes stop outputting/fill up buffers Graylog Central (peer support) Colleague of mine has subscribed to the Graylog newsletter. T…

I checked the thread dump. Could not see any blocking threads. All the inputs i use are Gelf/Kafka inputs. I have around 17 inputs reading from kafka topics for each particular inputs.

Having had a similar issue in my environment and looking at other posts over time I am beginning to suspect it is a runaway/recursive GROK issue. I have mostly eliminated GROK in favor of key/value, split and regex and is seems to have helped. Not sure yet as I never see any errors in logs.

Sorry. I should have mentioned this. I am not using any GROK patterns. All my inputs are GELF which is JSON format.

Which buffer is full first before logs fill the journal ? process buffer ? output buffer ? (you can see it in System -> Nodes, select a node) Which VM is limiting the throughput ? Graylog ? Elastic ? Which resource is limiting the throughput ? CPU ? Disk I/O ?

It is the process buffer that is always full. [image] Which VM is limiting the throughput? Graylog ? Elastic ? its Graylog. ES servers use less than 10% CPU power. Which VM is limiting the throughput ? Graylog ? Elastic ? Not sure how to get this. CPU is on almost 90% on the servers. Messages…

There was further research in the post I put out recently that helped show which messages were locking up the process buffers. It was definitely a GROK-lock scenario for me… It shows in the post where to look for locked process buffers. [image] Graylog nodes stop outputting/fill …

For CPU use the “top” command and check the load averages (the three values). For Disk I/O check with the command “iostat” and also on the ESX host. What is the hardware ? SSD ? RAID ?

Graylog Output issue to Elasticsearch

Graylog Central (peer support)

tmacgbay (Tmacgbay) April 17, 2020, 5:56pm 2

Perhaps related to this post where it was a GROK issue? the post at the end shows the beginnings of how it was tracked down…

Topic		Replies	Views
Graylog output will stop Graylog Central (peer support)	9	560	August 31, 2023
Buffer Configuration Issue Graylog Central (peer support)	7	2423	June 17, 2021
The log is too large to cause blockage, no output is written Graylog Central (peer support)	3	492	July 25, 2023
Issues with Graylog after moving to an elasticsearch cluster Graylog Central (peer support)	20	2742	June 10, 2018
Slow Message processing from GELF/Kafka inputs Graylog Central (peer support)	10	2257	April 23, 2018

Graylog Output issue to Elasticsearch

Related topics