Datanode not listening on 9200 - no opensearch

Graylog Central (peer support)
11

I’ve set this, but this does not change anything, as OpenSearch is not starting.

I’ve tried specifying these options, hoping it would make opensearch start:

  • transport_certificate
  • transport_certificate_password
  • http_certificate
  • http_certificate_password

But it didn’t.
That .p12 file is correctly parsed, but the datanode complains that opensearch is not started:

2025-02-17T09:49:01.304+01:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
datanode.log 
2025-02-17T09:48:59.658+01:00 INFO  [OpensearchDataDirCompatibilityCheck] Found 0 indices and all of them are valid with current opensearch version 2.15.0
2025-02-17T09:48:59.875+01:00 INFO  [DatanodeDirectories] Opensearch of the node 74f09692-f70d-4c11-997d-669854d690c8 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-02-17T09:49:00.927+01:00 INFO  [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 914.9 ms
2025-02-17T09:49:00.957+01:00 INFO  [ServerBootstrap] Graylog datanode 6.1.6+a644883 starting up
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-31-cloud-amd64
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] Deployment: deb
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] Arch: amd64
2025-02-17T09:49:00.979+01:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-02-17T09:49:00.979+01:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-02-17T09:49:00.980+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-17T09:49:00.985+01:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-17T09:49:00.990+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-17T09:49:01.000+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-17T09:49:01.006+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-17T09:49:01.077+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-17T09:49:01.250+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: graylog.<FQDN>, 172.28.14.4
2025-02-17T09:49:01.250+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following serial number: 622915052742269610821727451503118679977033641676
2025-02-17T09:49:01.251+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following validity: 2025-02-13T14:55:17.000+0100 - 2035-02-12T14:55:17.000+0100
2025-02-17T09:49:01.267+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: graylog.<FQDN>, 172.28.14.4
2025-02-17T09:49:01.268+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following serial number: 622915052742269610821727451503118679977033641676
2025-02-17T09:49:01.268+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following validity: 2025-02-13T14:55:17.000+0100 - 2035-02-12T14:55:17.000+0100
2025-02-17T09:49:01.270+01:00 INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-02-17T09:49:01.302+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2025-02-17T09:49:01.302+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
2025-02-17T09:49:01.303+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical].
2025-02-17T09:49:01.304+01:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[guava-33.3.1-jre.jar:?]
	at org.graylog.datanode.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:228) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.doRun(CmdLineTool.java:305) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.run(CmdLineTool.java:246) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
	Caused by: java.lang.NullPointerException: Cannot read the array length because "<parameter1>" is null
		at java.base/java.util.Arrays.stream(Unknown Source) ~[?:?]
		at org.graylog.datanode.configuration.TruststoreCreator.findRootCert(TruststoreCreator.java:101) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.TruststoreCreator.addRootCert(TruststoreCreator.java:65) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.variants.OpensearchSecurityConfiguration.configure(OpensearchSecurityConfiguration.java:92) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:159) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:218) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:95) ~[graylog-datanode.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[guava-33.3.1-jre.jar:?]
		at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.3.1-jre.jar:?]
		at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-17T09:49:01.312+01:00 INFO  [Server] SIGNAL received. Shutting down.
2025-02-17T09:49:01.314+01:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2025-02-17T09:49:01.314+01:00 INFO  [GracefulShutdown] Goodbye.