Hello,
we using Graylog for a while and we current have few hundred indices with about 120 millions documents in each index.
From some days we often see warnings in elasticsearch logs:
org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<http_request>] would be…
Currently graylog and elasticsearch works on one machine. And we are little over “magic 32GB java heap size” for elasticsearch.
We consider to split data with 2 elasticsearch nodes works in cluster (each node with less than 32 GB heap).
We think about hot-warm architecture like in this description: Elasticsearch Hot Warm Architecture | Elastic Blog
but I’m not sure how connect graylog to this.
Could you also tell me can Graylog manage “moving” indices from hot to warm node and manage retention on such cluster?
Maybe anyone had similar issue and know better solution than hot-warm cluster?
Thanks