I setup a graylog to collect kuberentes logs for few days, it works fine. the output thouthput is around 2500 msg/sec, I try to change the configuration, but it didn’t work. could you please give some advice.
version: 2.5.1-3 3 nodes: 6 core, 6 g java opts: -Djava.net.preferIPv4Stack=true -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms3000m -Xmx3000m
version: 6.7.0 3 master nodes, 3 data nodes data node: 2 core 8 g master node: 1 core 2 g
total have 41 streams, and rule type are “match exactly” and “contain”, no regext match rule. No pipeline
for alerting, I have setup 20 conditions.
GRAYLOG_OUTPUTBUFFER_PROCESSORS: 1 GRAYLOG_PROCESSBUFFER_PROCESSORS: 5 GRAYLOG_INPUTBUFFER_PROCESSORS: 1 GRAYLOG_ALLOW_HIGHLIGHTING: true GRAYLOG_OUTPUT_BATCH_SIZE: 1000 GRAYLOG_RING_SIZE: 262144 GRAYLOG_MESSAGE_JOURNAL_MAX_AGE: 48h
here is one of the gl node’s metrics
stream Matcher filter time
If I stop half of the stream, the thouthput can reach 4000 msg/sec.
And I checked this article https://www.graylog.org/post/back-to-basics-monitoring-graylog, but didn’t know how to decrease the processing time,and ddentify the bottleneck of the thoughput.
After change output batch size to 5000