My CPU’s are at about 70%, I have 2 GB RAM free and the swap is under 5%. My disks are fast.
But usually the IN msg/sec are greater than the OUT msg/sec, and so the journal is increased along the time…and finally I have to stop the input corresponding to syslog in order empty the journal, and after that start the same input.
I think my hardware is big enough because I can’t see any warning in the resources utilization.
What can I do in order to process all the incoming messages and don’t fill the journal ???
first, a little hint:
You should always set the -Xms and -Xmx values the same. This has to do with heap space allocation, a very costly process. (Also mentioned in the Elasticsearch article about memory)
And your server seems big enough to always spend those two extra gigs on startup
For your problem, 1 question and 1 suggestion:
What does your Elasticsearch look like (config etc.)? If the output does not perform well enough, it is the speed of elasticsearch or the speed of the connection between Graylog and Elasticsearch most of the time. Try the Extending Current Setup chapter of the Back to Basics: From Single Server to Graylog Cluster Graylog blog article. (Another helpful link: Elasticsearch Refresh Interval vs Indexing Performance)
depending on what you do with processing that given heap is way to much … lower the heap to ~4 GB that should be OK. But as @derPhlipsi said, the ruler in this game is your Elasticsearch (Cluster).
