There were 204,064 failed indexing attempts in the last 24 hours

Hi,
Graylog is not receiving any new messages, after click “rotate active write index” Elasticsearch turn the status into red with Elasticsearch cluster is red Shards: 92 active, 0 initializing, 0 relocating, 80 unassigned, please help

you should have 0 unassigned shards.
Check your elasticsearch cluster.
Also you should check graylog and elasticsearc’s logs.

I would like to suggest the following page, to help us with information to help you.

is it okay to delete unassigned in all node?

It depends on you.
Do you know what data do you have in the unassigned shards?
If you don’t need it, you can delete it. But I think it won’t solve the problem.

Here is some basic information about the indexing process in graylog.
Check it, I think ,it will clear some of your questions.
https://docs.graylog.org/en/3.1/pages/configuration/index_model.html

Check the disk space

The disk space of graylog is 10% elasticsearch data is 85%.
how do i clear unassigned or restore? did alredy rotate active write index

got this error in allocation explain:

"failed shard on node [KvhaB3zRTDOPaQG0TACx8A]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[graylog_25][3]: obtaining shard lock timed out after 5000ms]; ",
“last_allocation_status” : “no”
},
“can_allocate” : “no”,
“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy”,
“node_allocation_decisions” : [
{

“shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-01-15T06:38:52.229Z], failed_attempts[5], delayed=false, details[failed shard on node [KvhaB3zRTDOPaQG0TACx8A]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[graylog_25][3]: obtaining shard lock timed out after 5000ms]; ], allocation_status[deciders_no]]]”

elasticsearch status is now green but still there’s no new messages found in graylog the latest messages in indices are Jan.8, 2020

I’ve fixed the issue

that someone else is able to get something out of your posting - what was the issue and how did you solve this?

2 Likes

The issue (i assume) was the default setting for Allocation Watermark in cluster settings in elasticsearch.
By default, if it reaches 85% of disk space on elastic data node it stops shards from allocating to the same node. Possible solution is to raise the watermark up to 90% or add datanodes to elasticsearch cluster to allow proper reballancing (or, add disk space to all datanodes )

GET /_cluster/settings?include_defaults

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.