Elasticsearch optimization

Graylog Central (peer support)
2

Hi @danpop
welcome to the community. You have a quite big instance already, congratulations! I have a few thoughts:

  1. You have three nodes with Elastic with 64GB RAM each, and a Heap size of 32GB. Java has the “feature” to increase the RAM usage a lot if you supply it with 32GB RAM. Better go for double machines and half everything.
  2. your Elastic is a little older. If you upgrade make sure to find your path to Opensearch, as Graylog will use that in the long run.
  3. I gave once a very rough intro how to tune Elastic. Did you take care of the sizes of your shards?
  4. output_batch_size = 17000 is huge. I have I think 500 to 1000 at max.
  5. you have 1 inputbuffer_processor, 17 processbuffer_processors and 13 outputbuffer_processors. That are 31 in total. I understand if you not want to overprovision your CPU, but with 10-15% more I was fine so far.
  6. are all of your elastic-nodes in your graylog-config? If your graylog “talks” only to one of them it might be the bottleneck.
2 Likes