Hardware of this bare-metal install is - 32thread CPU and 64GB ram
i need proper config to allocate as flexibly as possible this resources.
Now im have stuck with this - all my extractors and pipelines makes my server huge pain cuz i see parsed messages only 1 hour old - others in a queue on parse and about 50million message average on utilization unprocessed messages
Is this hardware dedicated to Graylog or is does the Data Node and mongo also exist on this node?
What is the heap value assigned to Graylog?
Is the process buffer at 100% and the output buffer empty or is the output buffer also 100%?
It would also be worthwhile to see which streams/pipelines/rules are consuming the most compute. Could you expand on your approach to parsing data, for example, are you separating the logs into individual streams with pipelines attached to those streams so that only the related logs to that processing are being evaluated.
An aggregation like the one below might help give some insight.
then i created about 40 extractors which find a string CEF and then via regexp making extracting field
also i have about 10 platforms, which have unique source_ip. so i created on every platform own stream with typical rules - if source ip match - data go to this stream
===========
about pipelines - from all data that coursed on graylog - every message (which have string CEF) - checks via regexp to find a field srcIP, when it finds - makes new field srcIPP. simmilary makes with dstIP
and then i compared fields dstIP and srcIP - and on my rules - this go to specific stream/
its only 1 pipeline with 2 stages inside - check srcip dstip, and then compare values and then writes to stream
The fact that output buffer is 100% indicates the issue is with the indexer (Data Node) being unable to keep up with the load.
Generally it is preferred that the components that comprise Graylog (Graylog, Mongo and Data Node) are separated to avoid resource contention.
What heap value is set for Data Node? This will be found under the opensearch_heap option within the datanode.conf file. Also what is your current daily ingest in GB, this can be found under System/Overview.
Thanks for providing all this, the heap is as expected which is good to see. However, the ingest is very high for a single node to index.
Realistically, you would want an load balanced setup of 2 - 3 Graylog Nodes and 3 Data Nodes to handle 800GB of ingest per day. Is the current 800GB required are can you reduce this by dropping logs that are not of high value?
