Hi
I setup a graylog to collect kuberentes logs for few days, it works fine. the output thouthput is around 2500 msg/sec, I try to change the configuration, but it didn’t work. could you please give some advice.
Graylog server:
version: 2.5.1-3
3 nodes: 6 core, 6 g
java opts: -Djava.net.preferIPv4Stack=true -XX:NewRatio=1 -server -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled
-XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms3000m -Xmx3000m
ES server:
version: 6.7.0
3 master nodes, 3 data nodes
data node: 2 core 8 g
master node: 1 core 2 g
total have 41 streams, and rule type are “match exactly” and “contain”, no regext match rule. No pipeline
for alerting, I have setup 20 conditions.
custom configration:
GRAYLOG_OUTPUTBUFFER_PROCESSORS: 1
GRAYLOG_PROCESSBUFFER_PROCESSORS: 5
GRAYLOG_INPUTBUFFER_PROCESSORS: 1
GRAYLOG_ALLOW_HIGHLIGHTING: true
GRAYLOG_OUTPUT_BATCH_SIZE: 1000
GRAYLOG_RING_SIZE: 262144
GRAYLOG_MESSAGE_JOURNAL_MAX_AGE: 48h
here is one of the gl node’s metrics
stream Matcher filter time
If I stop half of the stream, the thouthput can reach 4000 msg/sec.
And I checked this article https://www.graylog.org/post/back-to-basics-monitoring-graylog, but didn’t know how to decrease the processing time,and ddentify the bottleneck of the thoughput.
After change output batch size to 5000
