So… Here are the buffers below in a screen clip for clarification. If there is a problem with storing things in Elasticsearch, the “Output Buffer” would fill up…because Graylog is trying to Output to ElasticSearch.
But you stated your “Process Buffer” was filling… which means the issue is in Graylog when it is processing the message before storing it in ElasticSearch for future searching. You can mess with ElasticSearch all you want but it doesn’t solve your issue as described. On an Elasticsearch side note, that article is very old and is several versions back on Graylog from the 4.2.13 you are running… there have been significant architecture changes since then.
As messages come into your Graylog Inputs, they are examined by Graylog (which stores it’s configurations in MongoDB) then modified - as commanded - before shipping out to Elasticsearch. Here is a link to community built Message Process Flow
You may want to reconsider how you are processing messages, regex/GROK can be terribly inefficient (AND EVEN LOCK UP YOUR PROCESSING BUFFER) if you are not careful. Some of what you have lightly described may have to iterate a message multiple times to hit a match… that is a LOT of wasted processing failures!
Here are my recommendations:
-
Create more inputs and shift things that you control to those “Unknown to Developers” ports so you have better control of what you can control. (Actually, this ideas belongs to @gsmith
) -
Move all of your extractors off the Developer Input in and put them in a pipeline.
-
Create pipeline rules for the Developer Input that, in the first stage, either truncate large messages or drop them all together. Use the aforementioned-in-the-post functions. If you move your extractors completely to pipelines you can re-use the rules you have for both the Secret Inputs and the Developer Input… After truncation/drop of course.
Hope that helps… 