Incident
After updating graylog server (open) and graylog datanode to 6.1.5 from 6.1.4 it will no longer connect for the web ui. Prior to this it was running without issue.
- Web UI was being accessed via IP over http.
- Logs have the server name redacted to “SERVER-FQDN” or “SERVER-NAME” depending on what is being referenced.
- OopenSearch is installed but it is not running, is disabled and not configured. GrayLog WAS working with it in this state (disabled) prior to the upgrade. Current behavour occurs wether it is running or not.
Environment
- OS Information: CentOS 7
Versions
- Graylog Server: v6.1.5-1
- Graylog Datanode: v6.1.5-1
- MongoDB: 6.0.19-1.rl7
- OpenSearch: v2.18.0-1 (disabled and not configured, installed by mistake)
Logs
Server Log
2025-01-13T08:59:07.371-08:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[cloud_inputs=on, investigation_report_by_ai=on, data_tiering_cloud=off, composable_index_templates=off, preflight_web=on, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, configurable_value_units=on, new_report_creation=on, threat_coverage=on]}
2025-01-13T08:59:07.983-08:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 6.1.5+e3ae3ce [org.graylog.aws.AWSPlugin]
2025-01-13T08:59:07.983-08:00 INFO [CmdLineTool] Loaded plugin: Integrations 6.1.5+e3ae3ce [org.graylog.integrations.IntegrationsPlugin]
2025-01-13T08:59:07.984-08:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 6.1.5+e3ae3ce [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-01-13T08:59:07.984-08:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 6.1.5+e3ae3ce [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-01-13T08:59:07.985-08:00 INFO [CmdLineTool] Loaded plugin: OpenSearch 2 Support 6.1.5+e3ae3ce [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-01-13T08:59:08.010-08:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=rpm
2025-01-13T08:59:08.218-08:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "3.10.0-1160.119.1.el7.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7fdab70c, com.mongodb.Jep395RecordCodecProvider@25ad4f71, com.mongodb.KotlinCodecProvider@49faf066]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-13T08:59:08.222-08:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "3.10.0-1160.119.1.el7.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7fdab70c, com.mongodb.Jep395RecordCodecProvider@25ad4f71, com.mongodb.KotlinCodecProvider@49faf066]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-13T08:59:08.244-08:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=21923541, minRoundTripTimeNanos=0}
2025-01-13T08:59:08.324-08:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 6.0.19
2025-01-13T08:59:08.760-08:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "3.10.0-1160.119.1.el7.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7fdab70c, com.mongodb.Jep395RecordCodecProvider@25ad4f71, com.mongodb.KotlinCodecProvider@49faf066]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-13T08:59:08.761-08:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1618302, minRoundTripTimeNanos=0}
2025-01-13T08:59:08.762-08:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "3.10.0-1160.119.1.el7.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7fdab70c, com.mongodb.Jep395RecordCodecProvider@25ad4f71, com.mongodb.KotlinCodecProvider@49faf066]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-13T08:59:09.619-08:00 INFO [ServerBootstrap] Running 2 migrations...
2025-01-13T08:59:09.952-08:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to SERVER-FQDN/127.0.1.1:9200. - Connection refused.
2025-01-13T08:59:09.954-08:00 INFO [VersionProbe] Indexer is not available. Retry #1
2025-01-13T08:59:14.956-08:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to SERVER-FQDN/127.0.1.1:9200. - Connection refused.
2025-01-13T08:59:14.957-08:00 INFO [VersionProbe] Indexer is not available. Retry #2
2025-01-13T08:59:19.959-08:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to SERVER-FQDN/127.0.1.1:9200. - Connection refused.
DataNode Log
2025-01-13T09:02:33.670-08:00 INFO [OpensearchProcessImpl] [2025-01-13T09:02:33,670][INFO ][o.o.n.Node ] [SERVER-FQDN] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8438311646608672137, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Xms1g, -Xmx1g, -Dopensearch.transport.cname_in_publish_address=true, -Djavax.net.ssl.trustStore=/var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12, -Djavax.net.ssl.trustStorePassword=YCIwOuNlAHkPrBlFRAfbuQfUYaPVFEJspHtzIyjJvbzoZLjVwxhoBNfPutYjzMxSgqlXGFFymdvfMREaVsbmjCBgYdvpUQfYJBVWtLdULBYMOcxABsSdQCHRPvSDOLynzzgwqCJuhheBUfkQcQjcvYOuagilCXdrAdOmGJKASBcZmtLybBZYbMbNevEjcYGvHtOFegHTXvdKEKtSYWfWYPYkHzVDpQZnlpZSvAxUSBwrYdqAchugLAqMyxolhTNs, -Djavax.net.ssl.trustStoreType=pkcs12, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
2025-01-13T09:02:33.773-08:00 INFO [OpensearchProcessImpl] [2025-01-13T09:02:33,761][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [SERVER-FQDN] uncaught exception in thread [main]
2025-01-13T09:02:33.774-08:00 INFO [OpensearchProcessImpl] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [opensearch-observability]
2025-01-13T09:02:33.774-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.774-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.774-08:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.774-08:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] Caused by: java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [opensearch-observability]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:466) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.findBundles(PluginsService.java:451) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.775-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.getPluginBundles(PluginsService.java:444) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:184) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.776-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] ... 6 more
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] Caused by: java.nio.file.NoSuchFileException: /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/plugins/opensearch-observability/plugin-descriptor.properties
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
2025-01-13T09:02:33.777-08:00 INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) ~[?:?]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at java.base/java.nio.file.Files.newByteChannel(Files.java:379) ~[?:?]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at java.base/java.nio.file.Files.newByteChannel(Files.java:431) ~[?:?]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at java.base/java.nio.file.Files.newInputStream(Files.java:159) ~[?:?]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:258) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:463) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.778-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.findBundles(PluginsService.java:451) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] uncaught exception in thread [main]
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.getPluginBundles(PluginsService.java:444) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [opensearch-observability]
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:184) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] Likely root cause: java.nio.file.NoSuchFileException: /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/plugins/opensearch-observability/plugin-descriptor.properties
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.779-08:00 WARN [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
2025-01-13T09:02:33.779-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
2025-01-13T09:02:33.780-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at java.base/java.nio.file.Files.newByteChannel(Files.java:379)
2025-01-13T09:02:33.780-08:00 INFO [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.15.0.jar:2.15.0]
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at java.base/java.nio.file.Files.newByteChannel(Files.java:431)
2025-01-13T09:02:33.780-08:00 INFO [OpensearchProcessImpl] ... 6 more
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at java.base/java.nio.file.Files.newInputStream(Files.java:159)
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:258)
2025-01-13T09:02:33.780-08:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.readPluginBundle(PluginsService.java:463)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.findBundles(PluginsService.java:451)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.getPluginBundles(PluginsService.java:444)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:184)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:498)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.node.Node.<init>(Node.java:425)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
2025-01-13T09:02:33.781-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.cli.Command.main(Command.java:101)
2025-01-13T09:02:33.782-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
2025-01-13T09:02:33.783-08:00 WARN [OpensearchProcessImpl] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
2025-01-13T09:02:33.783-08:00 WARN [OpensearchProcessImpl] For complete error details, refer to the log at /var/log/graylog-datanode/opensearch/datanode-cluster.log
2025-01-13T09:02:33.835-08:00 WARN [OpensearchProcessImpl] Opensearch process failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:355) ~[commons-exec-1.4.0.jar:1.4.0]
at org.apache.commons.exec.DefaultExecutor.lambda$execute$0(DefaultExecutor.java:269) ~[commons-exec-1.4.0.jar:1.4.0]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-01-13T09:02:33.836-08:00 INFO [OpensearchCommandLineProcess] Process 17754 still alive, waiting for termination. Retry #1
2025-01-13T09:02:33.836-08:00 INFO [OpensearchCommandLineProcess] Process 17754 successfully terminated.
2025-01-13T09:02:33.839-08:00 INFO [ClusterNodeStateTracer] Updating cluster node 7a86d40f-9c05-4337-a32c-745384d2bf13 from STARTING to UNAVAILABLE (reason: PROCESS_TERMINATED)
2025-01-13T09:02:33.842-08:00 WARN [OpensearchWatchdog] Process watchdog terminated after too many restart attempts
Configurations
server.conf
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = [SECRET]
root_password_sha2 = [ROOT_PASSWORD]
root_timezone = America/Los_Angeles
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.50.14.191:9000
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
transport_email_enabled = true
transport_email_hostname = [SMTP_RELAY]
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_from_email = [RETURN_EMAIL]
transport_email_socket_connection_timeout = 10s
transport_email_socket_timeout = 10s
integrations_scripts_dir = /usr/share/graylog-server/scripts
datanode.conf
node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = [SECRET]
root_password_sha2 = [ROOT_PASSWORD]
mongodb_uri = mongodb://localhost/graylog
bind_address = 0.0.0.0
opensearch_http_port = 9200
opensearch_transport_port = 9300
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
Is there something simple I am missing? I’m about to go scorched earth and burn it all and start over. Thanks!