I have used a shell to open a long TCP connection for send messages to a Graylog server in a test bed.
my problem is the output (to Elasticsearch) message rate(1.5K to 2K MPS) are not matched input message rate(12K to 13K MPS). My Elasticsearch cluster with 3 servers all are KVM virtual host, each with 2 cores and 24 GB RAM.
i am observing the improvements by adjust the resources to ES KVM hosts. but i want to know if your guys have any suggestion on this.
There are a few interesting articles linked in this GitHub issue:
https://github.com/Graylog2/documentation/issues/79
thanks jochen,
from the observing, the most busy one is the graylog server to which we input those messages, its CPU almost burst to 230%(and even after all messages posted, the graylog server keep a high CPU for a long time). and elasticsearch some time come up and down between nearly 40%- to 100%+. the message size average 540 bytes or so.
the CPU the machine comes with E5-2620@2.1Gz, each graylog with 4 cores, each es with 2 cores. their memory usage is stable at somewhere.
I have tried to remove the index replicas and rotate the active index and add 2 more cores to a graylog server which we input message, the output MPS to ES improved dramatically. but the CPU usage of the Graylog server remain very high and ES’ CPU now goes down under 100%.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
