Elasticsearch disconnecting frequently

Graylog Central (peer support)
1

Hello Friends,

My elasticsearch disconnecting with Graylog frequently after 3 or 4 days. I am getting below error, can you please let me know what could be the cause of these errors.

Failed to index message: index=<is02_1y_2> id=
error=<{“type”:“unavailable_shards_exception”,“reason”:"[is02_1y_2][1] primary shard is not active Timeout: [1m],
request: [BulkShardRequest [[is02_1y_2][1]] containing [47] requests]"}>
ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{

2

Hello,

Maybe I can help.
There are a couple commands you can excute to find out what, why, and how to solve this issue.

Check Elasticsearch Health it should state “Green”.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

The following command would indicate that there might be something wrong with the shards.(Look for Unassigned shards/Errors)

curl -XGET http://localhost:9200/_cat/shards | more

If errors were found from the previous command, the following commmand will find out why those error/s occured.

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

I looks like you Primary Shards are not active which tell me there might be something wrong with your elasticsearch. If you execute the above commands in order you might be able to identify or get close on whats going on in your setup.
Hope that helps

3

Hi gsmit,

Below commands executed , output are given below. please guide me to resolve this.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

“cluster_name” : “docker-cluster”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 104,
“active_shards” : 104,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0

curl -XGET http://localhost:9200/_cat/shards | more
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7592 100 7592 0 0 3602 0 0:00:02 0:00:02 --:–:-- 3601
graylog_2 2 p STARTED 4998396 1.6gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 5001780 1.6gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5000565 1.6gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 5000048 1.6gb 127.0.0.1 EnbqQkG
jirafrk-accesslogs-full_0 2 p STARTED 733480 287.6mb 127.0.0.1 EnbqQkG
jirafrk-accesslogs-full_0 1 p STARTED 733458 287.4mb 127.0.0.1 EnbqQkG
jirafrk-accesslogs-full_0 3 p STARTED 733589 287.8mb 127.0.0.1 EnbqQkG
jirafrk-accesslogs-full_0 0 p STARTED 733537 288.4mb 127.0.0.1 EnbqQkG
gl-system-events_1 0 p STARTED 0 261b 127.0.0.1 EnbqQkG
graylog_3 2 p STARTED 4998643 1.7gb 127.0.0.1 EnbqQkG
graylog_3 3 p STARTED 5001765 1.7gb 127.0.0.1 EnbqQkG
graylog_3 1 p STARTED 5000417 1.7gb 127.0.0.1 EnbqQkG
graylog_3 0 p STARTED 4999778 1.7gb 127.0.0.1 EnbqQkG
graylog_4 2 p STARTED 4999639 1.6gb 127.0.0.1 EnbqQkG
graylog_4 3 p STARTED 4997910 1.6gb 127.0.0.1 EnbqQkG
graylog_4 1 p STARTED 5000901 1.6gb 127.0.0.1 EnbqQkG
graylog_4 0 p STARTED 5001689 1.6gb 127.0.0.1 EnbqQkG
graylog_5 2 p STARTED 5003550 1.6gb 127.0.0.1 EnbqQkG
graylog_5 3 p STARTED 5000695 1.6gb 127.0.0.1 EnbqQkG
graylog_5 1 p STARTED 4997803 1.6gb 127.0.0.1 EnbqQkG
graylog_5 0 p STARTED 4998362 1.6gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 4997543 1.5gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 5001344 1.5gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5001422 1.5gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 4999975 1.5gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 5000135 1.6gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 5000403 1.6gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 4998487 1.6gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 5001279 1.6gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 4997481 1.7gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 5000845 1.7gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5002438 1.7gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 4999489 1.7gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 5000666 1.8gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 4999378 1.8gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5002349 1.8gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 4998137 1.8gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 4998782 1.6gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5001989 1.6gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 5001502 1.6gb 127.0.0.1 EnbqQkG
graylog_2 0 p STARTED 4998343 1.6gb 127.0.0.1 EnbqQkG
graylog_2 2 p STARTED 4999259 1.6gb 127.0.0.1 EnbqQkG
graylog_2 1 p STARTED 4998403 1.6gb 127.0.0.1 EnbqQkG
graylog_2 3 p STARTED 5003535 1.6gb 127.0.0.1 EnbqQkG

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
{
“error” : {
“root_cause” : [
{
“type” : “illegal_argument_exception”,
“reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]”
}
],
“type” : “illegal_argument_exception”,
“reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]”
},
“status” : 400
}

4

Hi gsmith,
Waiting for your revert, please let me know in case of other details required.

5

Hello,
I’m sorry for the delay, I’ve been working on learning more with Elasticsearch, which is taking all my time.
Thank you for the added details, it seems to look good so far.
Would you be able to show you Elasticsearch file ( elasticsearch.yml) and Graylog confgiuration file ( server.conf).

Simple command to make it easy to read these files when you post it. May something like this.

grep -v "^#\|^$" /etc/graylog/server/server.conf
and
grep -v "^#\|^$" /etc/elasticsearch/elasticsearch.yml

6

Thank you for your response.

As below command s are executed, output mentioned below.

grep -v “^#|^$” /usr/graylog/server/server.conf
[main]
no-auto-default=*
ignore-carrier=*

grep -v “^#|^$” /etc/elasticsearch/elasticsearch.yml
cluster.name: “docker-cluster”
network.host: 0.0.0.0

7

Hello,

What type of installation do you have? Your files do not look familiar nor correct.

EDIT: I show an example of my Graylog/Elasticsearch config files here.

8

[main]

no-auto-default=*
ignore-carrier=*
[/quote]

What type of installation do you have? Your files do not look familiar nor correct : It’s running as a container.

Yes, I have created containers by using the docker-compose file.

9

I thought this was a package installment. I dont think I’ll be able to help you since I’m really not familiar with Docker installment. Maybe someone here could help you better.
Sorry I can not be more help.

10

Did you already post your Docker-compose file? If not, that would be helpful.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.