Cluster setup on gralog

Development
1

i have graylog 7.0 and cluster set with 3 nodes the same for opensearch and mongod.
when i start http://1p:9000 for first time I have
While retrieving data for this widget, the following error(s) occurred:

  • OpenSearch exception [type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];].
    check with curl in all server(node) is ok:
    curl -X GET “http://10.9.71.120:9200”
    {
    “name” : “master-node-1”,
    “cluster_name” : “graylog”,
    “cluster_uuid” : “dvLowRQHQC-OdbF6RpsEbw”,
    “version” : {
    “distribution” : “opensearch”,
    “number” : “2.5.0”,
    “build_type” : “deb”,
    “build_hash” : “b8a8b6c4d7fc7a7e32eb2cb68ecad8057a4636ad”,
    “build_date” : “2023-01-18T23:48:43.426713304Z”,
    “build_snapshot” : false,
    “lucene_version” : “9.4.2”,
    “minimum_wire_compatibility_version” : “7.10.0”,
    “minimum_index_compatibility_version” : “7.0.0”
    },
    “tagline” : “The OpenSearch Project: https://opensearch.org/”
2

Hey,

Not sure Graylog have version 7 out, looks like your service is having issues. Check your

journalctl -xeu opensearch.

3

sorry i mean 6. i used opensearch version 2.5.0.
journalctl -xeu opensearch
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2028)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1891)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1325)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.node.Node.(Node.java:383)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.node.Node.(Node.java:356)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.cli.Command.main(Command.java:101)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Apr 23 08:25:09 graylog2 systemd-entrypoint[173]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Apr 23 08:25:20 graylog2 systemd[1]: Started OpenSearch.
░░ Subject: A start job for unit opensearch.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit opensearch.service has finished successfully.

4

Hey,

The journal logs you posted looks like the bottom half, you may need to scroll up. There should be a statement why it failed.

EDIT:

Also check the health of you indexer.

curl -X GET “http://10.9.71.120:9200_cluster/health"

status should be green

 "status" : "green",