we’re getting indexing failure every week on index set that has high traffic.
ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=no write index is defined for alias [xxx_deflector]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index]]
Every time, there are 2 indices marked as deflector. I can resolve the issue by removing the alias from one of the indices, but this is happening weekly.
I haven’t try to change the retention to 1 day yet. Surprisingly, it has been working fine for at least 1 week which was not the case. I will change the retention policy if it happens again. I would love to upgrade to 4.1 since it has native Prometheus support. Do you know from 4.0.8 to 4.1, do I also need to upgrade MongoDB and ElasticSearch? Below are the CPU/Memory/Disk IO
Thanks @gsmith. I’ve upgrade all my nodes to 4.1.7. I will let you know if the issue ever come back but so far it’s good (finger cross)
BTW, I’m also having another issue where all my nodes think they’re master node. I have 3 nodes cluster graylog-01, graylog-02, and graylog-03. I only specified is_master in the server.conf for graylog-01. Graylog Web is seeing all the nodes as master. I do see the a notification about multiple master servers everytime I restart any of the node.