Elasticsearch cluster is RED - Solved

Graylog Central (peer support)
1

Here is a partial output of http://172.20.32.95:9200/_cat/shards

moadmin.php            1 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
moadmin.php            1 r UNASSIGNED                                                
moadmin.php            3 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
moadmin.php            3 r UNASSIGNED                                                
moadmin.php            4 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
moadmin.php            4 r UNASSIGNED                                                
moadmin.php            2 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
moadmin.php            2 r UNASSIGNED                                                
moadmin.php            0 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
moadmin.php            0 r UNASSIGNED                                                
servlet                1 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
servlet                1 r UNASSIGNED                                                
servlet                3 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
servlet                3 r UNASSIGNED                                                
servlet                2 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
servlet                2 r UNASSIGNED                                                
servlet                4 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
servlet                4 r UNASSIGNED                                                
servlet                0 p STARTED         0   159b 172.20.32.95 Robert Bruce Banner 
servlet                0 r UNASSIGNED

Here is the output of http://172.20.32.95:9200/_cluster/health?pretty

{
  "cluster_name" : "graylog",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 244,
  "active_shards" : 244,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 244,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

So from what I can see each shard within each index has a primary and a replica. The replica is UNASSIGNED. For each index in the system, we see this same pattern. Is there a way to fix it?

I am new to graylog and it is a very sweet product. Thanks in advance for any help or suggestions!

Dave.

2

What exactly is the issue you’re trying to solve?

Regarding the state of your Elasticsearch cluster, I have the following observations:

  • The index names from the cat shards output look like you’re running Elasticsearch on a public network interface facing the Internet and someone performed an automatic scan against it which created these strange indices (“servlet”, “moadmin.php”).
  • With only 1 data node, having replicas doesn’t make sense.
3

I’m running Graylog and did a appliance installation. We have used it for a few months and why I raise the question is that Graylog is reporting Elasticsearch cluster is yellow. Shards: 4 active, 0 initializing, 0 relocating, 4 unassigned.

Does this answer your question?

4

You only have 1 Elasticsearch data node so that replica shards cannot be assigned anywhere.
By definition, this makes the cluster health state YELLOW (see http://docs.graylog.org/en/2.2/pages/configuration/elasticsearch.html#yellow).

Either reduce the number of replica shards to 0 or add (at least) another Elasticsearch node to your Elasticsearch cluster.

5

Look this.

6

Green!!!

Thanks guys! You are all very helpful. Hope you all have a great week.

1 Like
7

Just keep in mind that you have no resilience against node failures whatsoever if you only run 1 Elasticsearch data node and have no replicas.