OpenSearch Frozen - Graylog Docker

Graylog Central (peer support)
1

Graylog stopped working a few days ago because it was unable to connected to OpenSearch. My open search container appears to be frozen as I’m unable to connect to the docker console of that container. Not sure what else to say to help troubleshoot. Logs from the OpenSearch container below.

[2023-08-23T18:55:12,246][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [aggs-matrix-stats]
[2023-08-23T18:55:12,246][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [analysis-common]
[2023-08-23T18:55:12,246][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [geo]
[2023-08-23T18:55:12,246][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [ingest-common]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [ingest-geoip]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [ingest-user-agent]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [lang-expression]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [lang-mustache]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [lang-painless]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [mapper-extras]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [opensearch-dashboards]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [parent-join]
[2023-08-23T18:55:12,247][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [percolator]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [rank-eval]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [reindex]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [repository-url]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [search-pipeline-common]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [systemd]
[2023-08-23T18:55:12,248][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded module [transport-netty4]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-alerting]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-anomaly-detection]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-asynchronous-search]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-cross-cluster-replication]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-geospatial]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-index-management]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-job-scheduler]
[2023-08-23T18:55:12,249][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-knn]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-ml]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-neural-search]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-notifications]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-notifications-core]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-observability]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-performance-analyzer]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-reports-scheduler]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-security]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-security-analytics]
[2023-08-23T18:55:12,250][INFO ][o.o.p.PluginsService     ] [bb52fff0b890] loaded plugin [opensearch-sql]
[2023-08-23T18:55:12,283][INFO ][o.o.e.ExtensionsManager  ] [bb52fff0b890] ExtensionsManager initialized
[2023-08-23T18:55:12,291][DEPRECATION][o.o.d.c.s.Settings       ] [bb52fff0b890] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-08-23T18:55:12,304][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [bb52fff0b890] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/opensearch/data/nodes/1];
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.9.0.jar:2.9.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.9.0.jar:2.9.0]
uncaught exception in thread [main]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.9.0.jar:2.9.0]
Caused by: org.opensearch.OpenSearchException: failed to bind service
	at org.opensearch.node.Node.<init>(Node.java:1185) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.node.Node.<init>(Node.java:389) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.9.0.jar:2.9.0]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/opensearch/data/nodes/1
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
	at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
	at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]
	at org.opensearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:322) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:259) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.node.Node.<init>(Node.java:506) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.node.Node.<init>(Node.java:389) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.9.0.jar:2.9.0]
	... 6 more
OpenSearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/opensearch/data/nodes/1];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/opensearch/data/nodes/1
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
	at java.base/java.nio.file.Files.createDirectory(Files.java:700)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
	at java.base/java.nio.file.Files.createDirectories(Files.java:793)
	at org.opensearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:322)
	at org.opensearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:259)
	at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:320)
	at org.opensearch.node.Node.<init>(Node.java:506)
	at org.opensearch.node.Node.<init>(Node.java:389)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /usr/share/opensearch/logs/docker-cluster.log
2

Can you share more context about how you are using docker? Meaning is this running via docker desktop? What kind of host OS are you running docker on? Are all containers running on the same host?

Also can you also share your docker compose file?

It appears this error AccessDeniedException: /usr/share/opensearch/data/nodes/1 is saying that docker and opensearch cannot access that path. Have there been any changes to docker or the filesystem recently? Can you validate the permissions and ownership are correct?

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.