OpenSearch Issues / Search Dashboard empty

[jmayores (Jess)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Search dashboard is empty “Error retrieving data…”

2. Describe your environment:

• OS Information: Debian 12

• Package Version:

• Graylog 6.0.6

• Service logs, configurations, and environment variables:

Opensearch logs:

498]: index [graylog_4], id [1b137231-86d5-11ef-a4c9-0a3c2453a96f], message [OpenSearchException[OpenSearch exception [type=unavailable_shards_exception, reason=[graylog_4][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_4][0]] containing [500] requests]]]]
[499]: index [graylog_4], id [1b1dab60-86d5-11ef-a4c9-0a3c2453a96f], message [OpenSearchException[OpenSearch exception [type=unavailable_shards_exception, reason=[graylog_4][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_4][0]] containing [500] requests]]]]

3. What steps have you already taken to try and solve the problem?
Restart graylog service
Restart opensearch service
Rebooted the server

4. How can the community help?

Hi! I recently installed Graylog v6 (Community) and it is working very well. I hosted it in AWS EC2 running Debian 12.

Now I had observed that journal/messages and node/indices are getting bigger and bigger since I have a lot of devices and server on boarded.

So I decided to use AWS EFS service as the logs grows exponentially. Journal works Ok in EFS but my problem is the indices. What I did is to mount the EFS volume to the server and created a symlink (symbolic link) so that it will be pointed to EFS instead on the local server.

I performed rsync everything will be captured but I am getting the error in the dashboard

While retrieving data for this widget, the following error(s) occurred:

• OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed].

and these notifications:

Journal utilization is too high

(triggered 14 minutes ago)

Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Uncommited messages deleted from journal

(triggered 14 minutes ago)

Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit.

NOTE:
Graylog server and opensearch are started already but with those errors

Please help.

Thank you

[Wine_Merchant]

Hello @jmayores

The error “primary shard is not active Timeout” could mean a couple of different issues have occurred.

Are you certain there is enough space within the mounted EFS drive?

Did you update the location of the data store within your opensearch.yml?

What happens if you attempt to rotate the index from the Graylog UI?

[jmayores (Jess)]

Thanks for the reply.

There is space on the EFS and i had provisioned 1TB, it is only showing 100gb used.

I have this config:

Do i have to include the absolute path?

path.data: /var/lib/opensearch

Path to log files:

path.logs: /var/log/opensearch

indices is symlinked to this path:

indices → /mnt/efs/gl2-opensearch/nodes/0/indices/

Lastly, I rotated the index from the UI, it completed . but sill i get these errors:

Elasticsearch cluster graylog-2 is red. Shards: 0 active, 0 initializing, 0 relocating, 18 unassigned,

ournal utilization is too high

(triggered a few seconds ago)

Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit.

Thank you

[Wine_Merchant]

Does the opensearch user that runs the opensearch service have permissions on the mounted EFS drive (chown, chmod ect)?

What is the return of the below (alter the ip address)

curl -XGET ‘http://192.168.69.3:9200/_cluster/allocation/explain?pretty’

[jmayores (Jess)]

Yes opensearch user has permissions to the EFS directory

drwxr-xr-x 4 opensearch opensearch 6144 Oct 10 16:08 BgXLEII9TSWyr9QNZ7f4qQ
drwxr-xr-x 4 opensearch opensearch 6144 Oct 10 16:08 yCxx5hi7QvSirq1JOE2fmA
drwxr-xr-x 4 opensearch opensearch 6144 Oct 10 16:08 GE02CqRZRJ6GDBaDXEpEuA
drwxr-xr-x 4 opensearch opensearch 6144 Oct 10 16:08 G81SpLYMSnersiA5uSvY0g
root@aws-sg-gls-v3:/mnt/efs/gl2-opensearch/nodes/0/indices# pwd
/mnt/efs/gl2-opensearch/nodes/0/indices

Here is the result of the CURL:

root@aws-sg-gls-v3:/mnt/efs/gl2-opensearch/nodes/0/indices# curl -X GET “http://localhost:9200/_cluster/allocation/explain?pretty”
{
“index” : “mikrotik_0”,
“shard” : 0,
“primary” : true,
“current_state” : “unassigned”,
“unassigned_info” : {
“reason” : “ALLOCATION_FAILED”,
“at” : “2024-10-10T08:08:30.356Z”,
“failed_allocation_attempts” : 5,
“details” : “failed shard on node [fnDk7YVOTeK9dYb-Jcd3zw]: failed to create shard, failure AccessControlException[access denied ("java.io.FilePermission" "/mnt/efs/gl2-opensearch/nodes/0/indices/Ih3uunm2Q6KM24Xew8Fudw/0/index" "read")]”,
“last_allocation_status” : “no”
},
“can_allocate” : “no”,
“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes”,
“node_allocation_decisions” : [
{
“node_id” : “fnDk7YVOTeK9dYb-Jcd3zw”,
“node_name” : “aws-sg-gls-v3”,
“transport_address” : “127.0.0.1:9300”,
“node_attributes” : {
“shard_indexing_pressure_enabled” : “true”
},
“node_decision” : “no”,
“weight_ranking” : 1,
“deciders” : [
{
“decider” : “max_retry”,
“decision” : “NO”,
“explanation” : “shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-10-10T08:08:30.356Z], failed_attempts[5], failed_nodes[[fnDk7YVOTeK9dYb-Jcd3zw]], delayed=false, details[failed shard on node [fnDk7YVOTeK9dYb-Jcd3zw]: failed to create shard, failure AccessControlException[access denied ("java.io.FilePermission" "/mnt/efs/gl2-opensearch/nodes/0/indices/Ih3uunm2Q6KM24Xew8Fudw/0/index" "read")]], allocation_status[deciders_no]]]”
}
]
}
]
}

[jmayores (Jess)]

Anyone who has an idea?