The stacktrace is entirely different now:
datanode.log
2025-02-26T10:21:56.120+01:00 INFO [OpensearchProcessService] OpenSearch starting up
2025-02-26T10:21:56.121+01:00 INFO [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=1, OpensearchProcessService [RUNNING]=3, PeriodicalsService [RUNNING]=153, OpensearchConfigurationService [RUNNING]=522}
2025-02-26T10:21:56.121+01:00 INFO [ServerBootstrap] Graylog DataNode datanode up and running.
2025-02-26T10:21:56.126+01:00 INFO [OpensearchCommandLineProcess] Creating opensearch keystore
2025-02-26T10:21:56.130+01:00 INFO [JerseyService] Starting Data node REST API
2025-02-26T10:21:56.132+01:00 INFO [JerseyService] Jersey is using keystore located in /var/lib/graylog-datanode/opensearch/config/opensearch/http-keystore.p12
2025-02-26T10:21:56.709+01:00 INFO [Version] HV000001: Hibernate Validator 8.0.1.Final
2025-02-26T10:21:56.951+01:00 INFO [NetworkListener] Started listener bound to [127.0.0.1:8999]
2025-02-26T10:21:56.953+01:00 INFO [HttpServer] [HttpServer] Started.
2025-02-26T10:21:56.953+01:00 INFO [JerseyService] Started REST API at <127.0.0.1:8999>
2025-02-26T10:21:57.439+01:00 INFO [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch.keystore
2025-02-26T10:22:00.749+01:00 INFO [OpensearchCommandLineProcess] Added 4 keystore items
2025-02-26T10:22:00.754+01:00 INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/bin/opensearch
2025-02-26T10:22:00.838+01:00 INFO [ClusterNodeStateTracer] Updating cluster node f77dcc54-cfbb-4122-9d06-939130e9dd65 from UNCONFIGURED to STARTING (reason: PROCESS_STARTED)
2025-02-26T10:22:01.972+01:00 WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2025-02-26T10:22:01.979+01:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/opensearch-2.15.0.jar)
2025-02-26T10:22:01.980+01:00 WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
2025-02-26T10:22:01.980+01:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2025-02-26T10:22:02.435+01:00 WARN [OpensearchProcessImpl] Feb 26, 2025 10:22:02 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
2025-02-26T10:22:02.436+01:00 WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
2025-02-26T10:22:02.774+01:00 WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2025-02-26T10:22:02.775+01:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/opensearch-2.15.0.jar)
2025-02-26T10:22:02.775+01:00 WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
2025-02-26T10:22:02.775+01:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2025-02-26T10:22:02.790+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:02,784][INFO ][o.o.n.Node ] [localhost] version[2.15.0], pid[13439], build[tar/61dbcd0795c9bfe9b81e5762175414bc38bbcadf/2024-06-20T03:26:49.193630411Z], OS[Linux/6.1.0-30-cloud-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
2025-02-26T10:22:02.791+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:02,790][INFO ][o.o.n.Node ] [localhost] JVM home [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk], using bundled JDK/JRE [true]
2025-02-26T10:22:02.792+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:02,791][INFO ][o.o.n.Node ] [localhost] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5750266127598157313, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Xms1g, -Xmx1g, -Dopensearch.transport.cname_in_publish_address=true, -Djavax.net.ssl.trustStore=/var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12, -Djavax.net.ssl.trustStorePassword=GVocqKvXeTIepCRqBSTtwKXhwxnZuaPJMAOkYyOaPisoSqFuFDlThErOvQwzmKNHtjaCDcdHGYUJvtFAAAwVDCraCESkkaVwheIdHNPskWDSFkiJPMpiNzASOCwELGRtOuqYsMHJfjrchspPYFwxbFsHyfGgtsYQUYJCnYpmBDVENtOppNftbtvtcnYcVsWTDvsQfDZxUPmRPncyLPRPbcSqESoRicvmEcCrWXCzVKPZWxVjXSixqZlcXewssTUU, -Djavax.net.ssl.trustStoreType=pkcs12, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
2025-02-26T10:22:03.668+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,668][INFO ][o.o.s.s.t.SSLConfig ] [localhost] SSL dual mode is disabled
2025-02-26T10:22:03.670+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,670][INFO ][o.o.s.OpenSearchSecurityPlugin] [localhost] OpenSearch Config path is /var/lib/graylog-datanode/opensearch/config/opensearch
2025-02-26T10:22:03.878+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,877][INFO ][o.o.s.s.DefaultSecurityKeyStore] [localhost] JVM supports TLSv1.3
2025-02-26T10:22:03.882+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,879][INFO ][o.o.s.s.DefaultSecurityKeyStore] [localhost] Config directory is /var/lib/graylog-datanode/opensearch/config/opensearch/, from there the key- and truststore files are resolved relatively
2025-02-26T10:22:03.959+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,959][WARN ][o.o.s.s.u.SSLCertificateHelper] [localhost] Certificate chain for alias datanode contains a root certificate
2025-02-26T10:22:03.987+01:00 INFO [OpensearchProcessImpl] [2025-02-26T10:22:03,985][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [localhost] uncaught exception in thread [main]
2025-02-26T10:22:03.987+01:00 INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
2025-02-26T10:22:03.989+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.990+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.990+01:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.990+01:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.991+01:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.991+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.991+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.991+01:00 INFO [OpensearchProcessImpl] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
2025-02-26T10:22:03.991+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:803) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.992+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.995+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.996+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.996+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.996+01:00 INFO [OpensearchProcessImpl] ... 6 more
2025-02-26T10:22:03.997+01:00 INFO [OpensearchProcessImpl] Caused by: java.lang.reflect.InvocationTargetException
2025-02-26T10:22:03.997+01:00 INFO [OpensearchProcessImpl] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
2025-02-26T10:22:03.997+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
2025-02-26T10:22:03.998+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
2025-02-26T10:22:03.999+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.999+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.999+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.999+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:03.999+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.001+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.001+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.002+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.002+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.002+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.002+01:00 INFO [OpensearchProcessImpl] ... 6 more
2025-02-26T10:22:04.006+01:00 INFO [OpensearchProcessImpl] Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer: OpenSearchException[No certificates found in /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 with alias datanode]
2025-02-26T10:22:04.006+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:430) ~[?:?]
2025-02-26T10:22:04.007+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
2025-02-26T10:22:04.007+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:204) ~[?:?]
2025-02-26T10:22:04.007+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:252) ~[?:?]
2025-02-26T10:22:04.007+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:315) ~[?:?]
2025-02-26T10:22:04.007+01:00 INFO [OpensearchProcessImpl] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
2025-02-26T10:22:04.008+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
2025-02-26T10:22:04.008+01:00 WARN [OpensearchProcessImpl] uncaught exception in thread [main]
2025-02-26T10:22:04.008+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.009+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] ... 6 more
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] Caused by: org.opensearch.OpenSearchException: No certificates found in /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 with alias datanode
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.util.CertFromKeystore.validate(CertFromKeystore.java:94) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.util.CertFromKeystore.<init>(CertFromKeystore.java:60) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:407) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:204) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:252) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:315) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
2025-02-26T10:22:04.010+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-02-26T10:22:04.011+01:00 INFO [OpensearchProcessImpl] ... 6 more
2025-02-26T10:22:04.014+01:00 WARN [OpensearchProcessImpl] java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
2025-02-26T10:22:04.014+01:00 WARN [OpensearchProcessImpl] Likely root cause: OpenSearchException[No certificates found in /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 with alias datanode]
2025-02-26T10:22:04.014+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.util.CertFromKeystore.validate(CertFromKeystore.java:94)
2025-02-26T10:22:04.014+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.util.CertFromKeystore.<init>(CertFromKeystore.java:60)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:407)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:204)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:252)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:315)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794)
2025-02-26T10:22:04.015+01:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
2025-02-26T10:22:04.016+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
2025-02-26T10:22:04.017+01:00 WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/datanode-cluster.log
2025-02-26T10:22:04.042+01:00 WARN [OpensearchProcessImpl] Opensearch process failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:355) ~[commons-exec-1.4.0.jar:1.4.0]
at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:269) ~[commons-exec-1.4.0.jar:1.4.0]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-02-26T10:22:04.046+01:00 INFO [OpensearchCommandLineProcess] Process 13439 still alive, waiting for termination. Retry #1
The SSL certificate seems to be correctly passed to OpenSearch:
[2025-02-26T10:22:03,959][WARN ][o.o.s.s.u.SSLCertificateHelper] [localhost] Certificate chain for alias datanode contains a root certificate
But then the error is
Likely root cause: OpenSearchException[No certificates found in /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 with alias datanode]
Apparently, with an empty password I can list the contents:
sudo keytool -list -v -keystore /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 -storetype PKCS12
Enter keystore password:
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: datanode
Creation date: Feb 26, 2025
Entry type: PrivateKeyEntry
Certificate chain length: 0
*******************************************
*******************************************
So there is an entry datanode.