Down cluster with 3 node

[Uporaba (Yaroslav)]

Hello. I have a cluster of 3 nodes with elastic (2 node with graylog, and 1 single node with elastic), when one falls, the entire cluster fails (and becomes red). How can I fix this so that when one node fails, the rest work? My elastic settings:

cluster.name: graylog
node.name: es-node-01
#node.attr.rack: r1
path.data:
   - /var/lib/elasticsearch/data1
   - /var/lib/elasticsearch/data2
path.logs: /var/log/elasticsearch
#bootstrap.memory_lock: true
network.host: 10.101.15.231
http.port: 9200
discovery.seed_hosts: ["10.101.15.231", "10.101.15.232", "10.101.15.233"]
discovery.zen.fd.ping_timeout: 10s
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes:
  - es-node-01
  - es-node-02
  - es-node-03
#gateway.recover_after_nodes: 3
#action.destructive_requires_name: true
action.auto_create_index: false
node.roles: [ data, master ]
cluster.routing.allocation.disk.watermark.low: "20G"
cluster.routing.allocation.disk.watermark.high: "10G"
cluster.routing.allocation.disk.watermark.flood_stage: "5G"
path.repo:
   - /var/lib/elasticsearch/data1/backup

[gsmith]

Hello,

By chance is this related to this post.

Elasticsearch exception reason=all shards failed

hi, in the new version 4.1.3, this error appears. How can I understand what this error is and remove it? I don’t see any errors in the logs. [image] [image] but in graylog version 3.2.4 everything is searched and there are no errors: [image] UPD: In log i saw: 2021-09-28T07:50:04.217Z ERROR [MessagesAdapterES7] Failed to index [3] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution: [150]: index [graylog_57], type [_doc], id [b0d…

As for you question.

@Uporaba

How can I fix this so that when one node fails, the rest work? My elastic settings:

You might want to read this documentation on Multi-node Setup.

https://docs.graylog.org/en/4.1/pages/configuration/multinode_setup.html#configure-multinode

Also this documentation is very helpful.

Designing for resilience | Elasticsearch Guide [8.11] | Elastic

Hope that helps

[Uporaba (Yaroslav)]

@gsmith

By chance is this related to this post.

This is my post, but they are not related in any way, these are two different tasks

@gsmith

You might want to read this documentation on Multi-node Setup.

I read the documentation, but for some reason nothing works. I made the elastic cluster setup as shown in the official documentation (if I understood it correctly):

cluster.name: graylog
node.name: es-node-01
#node.attr.rack: r1
path.data:
   - /var/lib/elasticsearch/data1
   - /var/lib/elasticsearch/data2
path.logs: /var/log/elasticsearch
#bootstrap.memory_lock: true
network.host: 10.101.15.231
http.port: 9200
discovery.seed_hosts: ["10.101.15.231", "10.101.15.232", "10.101.15.233"]
discovery.zen.fd.ping_timeout: 10s
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes:
  - es-node-01
  - es-node-02
  - es-node-03
#gateway.recover_after_nodes: 3
#action.destructive_requires_name: true
action.auto_create_index: false
node.roles: [ data, master ]
cluster.routing.allocation.disk.watermark.low: "20G"
cluster.routing.allocation.disk.watermark.high: "10G"
cluster.routing.allocation.disk.watermark.flood_stage: "5G"
path.repo:
   - /var/lib/elasticsearch/data1/backup

the configuration is the same on all nodes

[Uporaba (Yaroslav)]

Is solved. I just needed add replics in index.

[gsmith]

Nice, and thanks for posting how you fixed it. This will help other down the road that may have the same problem/s.