Graylog + Opensearch cluster error

Jon_Doe · July 28, 2025, 12:01pm

Hello, so I have a problem with OpenSearch cluster. It says " OpenSearch cluster datanode-cluster is red. Shards: 87 active, 0 initializing, 0 relocating, 11 unassigned".

On “Streams > All events” tab it’s also returning error " While retrieving data for this widget, the following error(s) occurred:

OpenSearch exception [type=null_pointer_exception, reason=Cannot invoke “org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()” because “reducePhase.aggregations” is null]."

OS Information:
Oracle Linux Server 8.10
Opensearch version: 2.15.0
MongoDB ver. 2.5.5

Any suggestions?

patrickmann · July 28, 2025, 4:34pm

What version of Graylog?

Jon_Doe · July 29, 2025, 5:29am

Sorry, graylog ver. 6.1.12+23f653e, codename Noir

Wine_Merchant · July 29, 2025, 8:33am

hey @Jon_Doe,

Could you run the below api calls (altering the hostname and ensuring you have generated certs) and post the result?

curl -X GET --key key_data.crt --cert cert_data.crt --cacert ca_data.crt -u :password -H 'Content-Type: application/json' https://node-name:9200/_cat/shards?=pretty

curl -X GET --key key_data.crt --cert cert_data.crt --cacert ca_data.crt -u :password -H 'Content-Type: application/json' https://node-name:9200/_cat/indices?=pretty

Jon_Doe · July 29, 2025, 10:17am

I have a self-signed certificate and the results are:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I read the manual and tried the following:
Tried with “-v” and the result is:

Trying ((IP))
TCP_NODELAY set
Connected to graylog ((IP)) port 9200 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: tomato-ca.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, [no content] (0):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Request CERT (13):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (OUT), TLS alert, unknown CA (560):
SSL certificate problem: unable to get local issuer certificate
Closing connection 0

I’m still not sure how to fix it

Wine_Merchant · July 29, 2025, 10:56am

In order to query the Opensearch API you would need to generate a client cert from within the Graylog UI under system/cluster management and then the “Certificate Management” tab. Use the “Generate client certificate” option.

Jon_Doe · July 29, 2025, 12:09pm

Ok, so I’ve generated the certificates and created them manually on linux. I’m executing the command again with newly created files, but it seems i have to do something else:

curl: (51) SSL: no alternative certificate subject name matches target host name ‘graylog.DOMAIN.NAME’

Wine_Merchant · July 29, 2025, 12:14pm

The error suggests you are not using the hostname of the Data Node, under System/Cluster Management you should be able to see the hostname of your Data Node. Try that in the curl.

Jon_Doe · July 29, 2025, 12:37pm

Seems that it worked, but errors still occur. However this is the result:

.opendistro_security                                    0 p STARTED          10  76.1kb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.03-000120 0 p STARTED                     127.0.0.1 localhost
.ds-gl-datanode-metrics-000012                          0 p STARTED      376092  67.3mb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.05-000122 0 p STARTED                     127.0.0.1 localhost
.ds-gl-datanode-metrics-000010                          0 p STARTED       25803   5.1mb 127.0.0.1 localhost
.ds-gl-datanode-metrics-000011                          0 p STARTED        4270   1.5mb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.07-000124 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.09-000126 0 p STARTED                     127.0.0.1 localhost
.ds-gl-datanode-metrics-000003                          0 p STARTED        6398   1.4mb 127.0.0.1 localhost
.ds-gl-datanode-metrics-000004                          0 p STARTED       13840   2.8mb 127.0.0.1 localhost
gl-system-events_8                                      0 p STARTED           2  12.6kb 127.0.0.1 localhost
.ds-gl-datanode-metrics-000001                          0 p STARTED          92 133.1kb 127.0.0.1 localhost
gl-system-events_9                                      0 p STARTED           2  23.1kb 127.0.0.1 localhost
.ds-gl-datanode-metrics-000002                          0 p STARTED        3244 674.6kb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.14-000130 0 p STARTED                     127.0.0.1 localhost
gl-system-events_2                                      0 p UNASSIGNED
gl-system-events_3                                      0 p UNASSIGNED
.ds-gl-datanode-metrics-000009                          0 p STARTED        2879   1.1mb 127.0.0.1 localhost
gl-system-events_0                                      0 p UNASSIGNED
gl-system-events_1                                      0 p UNASSIGNED
.opendistro-ism-managed-index-history-2025.07.17-000133 0 p STARTED                     127.0.0.1 localhost
gl-system-events_6                                      0 p STARTED           2  12.6kb 127.0.0.1 localhost
.ds-gl-datanode-metrics-000007                          0 p STARTED       19738   3.3mb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.18-000134 0 p STARTED                     127.0.0.1 localhost
gl-system-events_7                                      0 p STARTED           2  12.6kb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.01-000118 0 p STARTED                     127.0.0.1 localhost
.ds-gl-datanode-metrics-000008                          0 p STARTED       17301   3.3mb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.02-000119 0 p STARTED                     127.0.0.1 localhost
.ds-gl-datanode-metrics-000005                          0 p STARTED       17303   3.7mb 127.0.0.1 localhost
gl-system-events_4                                      0 p UNASSIGNED
.ds-gl-datanode-metrics-000006                          0 p STARTED        3392 742.2kb 127.0.0.1 localhost
gl-system-events_5                                      0 p UNASSIGNED
.opendistro-ism-managed-index-history-2025.07.29-000145 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.06-000123 0 p STARTED                     127.0.0.1 localhost
.opendistro-job-scheduler-lock                          0 p STARTED          23  48.2kb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.13-000129 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.06.29-000116 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.25-000141 0 p STARTED                     127.0.0.1 localhost
graylog_25                                              0 p UNASSIGNED
graylog_24                                              0 p STARTED    22765666  16.3gb 127.0.0.1 localhost
graylog_27                                              0 p UNASSIGNED
graylog_29                                              0 p STARTED    22555269  15.5gb 127.0.0.1 localhost
graylog_28                                              0 p STARTED    22246613  15.7gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.22-000138 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.21-000137 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.20-000136 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.23-000139 0 p STARTED                     127.0.0.1 localhost
graylog_30                                              0 p STARTED    28219730  19.5gb 127.0.0.1 localhost
graylog_32                                              0 p STARTED    23345103  15.6gb 127.0.0.1 localhost
graylog_31                                              0 p STARTED    26939066  18.5gb 127.0.0.1 localhost
graylog_34                                              0 p STARTED    26869224  18.6gb 127.0.0.1 localhost
graylog_33                                              0 p STARTED    27777788  18.8gb 127.0.0.1 localhost
graylog_36                                              0 p STARTED    25590725  17.9gb 127.0.0.1 localhost
graylog_35                                              0 p STARTED    25396071  17.5gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.28-000144 0 p STARTED                     127.0.0.1 localhost
graylog_38                                              0 p STARTED    22290096  15.3gb 127.0.0.1 localhost
graylog_37                                              0 p STARTED    27799321  19.6gb 127.0.0.1 localhost
graylog_39                                              0 p STARTED    21715131  15.2gb 127.0.0.1 localhost
.opendistro-ism-config                                  0 p STARTED                     127.0.0.1 localhost
graylog_40                                              0 p STARTED    16377598  11.7gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.24-000140 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.12-000128 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.26-000142 0 p STARTED                     127.0.0.1 localhost
graylog_9                                               0 p STARTED    28747105  19.4gb 127.0.0.1 localhost
graylog_8                                               0 p STARTED    29059514  19.6gb 127.0.0.1 localhost
graylog_7                                               0 p UNASSIGNED
graylog_6                                               0 p STARTED    22228707  15.3gb 127.0.0.1 localhost
graylog_1                                               0 p STARTED    25784097  17.8gb 127.0.0.1 localhost
graylog_0                                               0 p UNASSIGNED
.opendistro-ism-managed-index-history-2025.07.15-000131 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.16-000132 0 p STARTED                     127.0.0.1 localhost
graylog_10                                              0 p STARTED    26754847  18.8gb 127.0.0.1 localhost
graylog_5                                               0 p STARTED    25077573  16.8gb 127.0.0.1 localhost
graylog_4                                               0 p STARTED    23404912  15.6gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.19-000135 0 p STARTED                     127.0.0.1 localhost
graylog_12                                              0 p STARTED    22057572  15.5gb 127.0.0.1 localhost
graylog_3                                               0 p STARTED    22514593  15.1gb 127.0.0.1 localhost
graylog_11                                              0 p STARTED    26417563  18.9gb 127.0.0.1 localhost
graylog_2                                               0 p STARTED    26135340  17.7gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.27-000143 0 p STARTED                     127.0.0.1 localhost
graylog_14                                              0 p STARTED    22964261    16gb 127.0.0.1 localhost
graylog_13                                              0 p STARTED    25986602  18.1gb 127.0.0.1 localhost
graylog_16                                              0 p STARTED    22771593  15.5gb 127.0.0.1 localhost
graylog_15                                              0 p STARTED    29123080  19.5gb 127.0.0.1 localhost
graylog_18                                              0 p STARTED    29033523  19.9gb 127.0.0.1 localhost
graylog_17                                              0 p STARTED    22364122  15.4gb 127.0.0.1 localhost
graylog_19                                              0 p STARTED    22881567  15.7gb 127.0.0.1 localhost
gl-events_1                                             0 p STARTED           0    208b 127.0.0.1 localhost
gl-events_0                                             0 p UNASSIGNED
.plugins-ml-config                                      0 p STARTED           1   3.9kb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.04-000121 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.06.30-000117 0 p STARTED                     127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.08-000125 0 p STARTED                     127.0.0.1 localhost
graylog_21                                              0 p STARTED    22484553  15.7gb 127.0.0.1 localhost
graylog_20                                              0 p STARTED    23008119  15.8gb 127.0.0.1 localhost
.opendistro-ism-managed-index-history-2025.07.11-000127 0 p STARTED                     127.0.0.1 localhost
graylog_23                                              0 p STARTED    22590856    16gb 127.0.0.1 localhost
graylog_22                                              0 p STARTED    27946184  19.7gb 127.0.0.1 localhost

drewmiranda-gl · July 29, 2025, 2:01pm

Can you share the output of? This will show a reason why the shards are unassigned.

curl -X GET --key key_data.crt --cert cert_data.crt --cacert ca_data.crt -u :password https://node-name:9200/_cluster/allocation/explain

Thanks!

Jon_Doe · July 30, 2025, 5:40am

{"index":"gl-system-events_2","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2025-07-29T12:29:36.118Z","last_allocation_status":"no_valid_shard_copy"},"can_allocate":"no_valid_shard_copy","allocate_explanation":"cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster","node_allocation_decisions":[{"node_id":"((ID))","node_name":"localhost","transport_address":"((IP)):9300","node_attributes":{"shard_indexing_pressure_enabled":"true"},"node_decision":"no","store":{"found":false}}]}

So, is there an option to keep the data from those shards?

Wine_Merchant · July 30, 2025, 8:15am

Has the Opensearch cluster only ever been made up of 1 node, could there be an issue with one of the disks that comprises the underlying storage?

Jon_Doe · July 30, 2025, 10:44am

I’ve tried to add one additional node but it ended up not working so I’ve deleted it. Maybe I forgot to clear something, I’ll look into that

Wine_Merchant · July 31, 2025, 12:42pm

Could it be the shards/indices that are now missing had been stored on the datastore of the now deleted node.

Jon_Doe · August 5, 2025, 7:13am

Ok so, I wasn’t able to find any old files,

BUT I double-checked unassigned shards using curl ……. “https://localhost:9200/\_cat/shards?v&h=index,shard,prirep,state,unassigned.reason”

and then curl ……….. “https://localhost:9200/\_cat/indices?v”

and I overlooked that my missing 11 shards were empty, so I’ve used this:

EXAMPLE

curl -X POST -k -u admin:admin --cert cert.crt --key key.key --cacert ca.crt “https://localhost:9200/_cluster/reroute” -H ‘Content-Type: application/json’ -d ‘{“commands”: [{“allocate_empty_primary”: {“index”: “gl-events_0”,“shard”: 0,“node”: “localhost”,“accept_data_loss”: true}}]}’

and I had to do this for every shard. It’s working now, thank you guys

Wine_Merchant · August 5, 2025, 8:21am

Nice work and thanks for sharing the solution @Jon_Doe!

system · August 19, 2025, 8:22am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch service is running but the Cluster is Red on the Web interface Graylog Central (peer support)	22	16258	November 17, 2017
Type=index_not_found_exception Graylog Central (peer support) elastic	19	10796	February 22, 2023
Moving a single opensearch to a cluster Graylog Central (peer support)	5	1475	January 16, 2024
Graylog Elasticsearch cluster is yellow since 3 days back Graylog Central (peer support)	10	3173	July 5, 2018
Graylog running but elasticsearch cluster health is red Graylog Central (peer support)	4	2953	March 1, 2018

Graylog + Opensearch cluster error

Related topics