1. Describe your incident:
Upgraded my 5.x instance to 6.x and now Opensearch 2.13 no longer starting
secdoc@cerebro:~$ sudo systemctl status opensearch.service
[sudo] password for secdoc:
× opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Sat 2024-05-25 18:12:54 UTC; 23min ago
Docs: https://opensearch.org/
Process: 8021 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status=143)
Main PID: 8021 (code=exited, status=143)
CPU: 18.839s
May 25 18:11:42 cerebro systemd-entrypoint[8021]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
May 25 18:11:42 cerebro systemd-entrypoint[8021]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
May 25 18:11:42 cerebro systemd-entrypoint[8021]: at org.opensearch.cli.Command.main(Command.java:101)
May 25 18:11:42 cerebro systemd-entrypoint[8021]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
May 25 18:11:42 cerebro systemd-entrypoint[8021]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
May 25 18:11:42 cerebro systemd-entrypoint[8021]: For complete error details, refer to the log at /var/log/opensearch/graylog.log
May 25 18:12:53 cerebro systemd[1]: opensearch.service: start operation timed out. Terminating.
May 25 18:12:54 cerebro systemd[1]: opensearch.service: Failed with result 'timeout'.
May 25 18:12:54 cerebro systemd[1]: Failed to start OpenSearch.
May 25 18:12:54 cerebro systemd[1]: opensearch.service: Consumed 18.839s CPU time.
All other services are running.
2. Describe your environment:
OS: Ubuntu 22.04.4 LTS x86_64
Kernel: 5.15.0-107-generic
CPU: AMD Ryzen 9 6900HX with Radeon Graphics (16) @ 3.300GHz
GPU: AMD ATI 05:00.0 Rembrandt
Memory: 952MiB / 28826MiB
Disk (/): 513G / 1.9T (29%)
secdoc@cerebro:~$ sudo apt list --installed | grep 'mongo\|elasticsearch\|opensearch\|graylog'
graylog-6.0-repository/stable,now 1-1 all [installed]
graylog-server/stable,now 6.0.2-1 amd64 [installed]
libmongoc-1.0-0/jammy,now 1.21.0-1build1 amd64 [installed,automatic]
libmongocrypt0/jammy,now 1.3.0-1ubuntu1 amd64 [installed,automatic]
mongodb-database-tools/jammy,now 100.9.4 amd64 [installed,automatic]
mongodb-mongosh/jammy,now 1.9.0 amd64 [installed,upgradable to: 2.2.6]
mongodb-org-database-tools-extra/jammy,now 6.0.15 amd64 [installed,automatic]
mongodb-org-database/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.15]
mongodb-org-mongos/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.15]
mongodb-org-server/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.15]
mongodb-org-shell/jammy,now 6.0.15 amd64 [installed,automatic]
mongodb-org-tools/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.15]
mongodb-org/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.15]
opensearch/stable,now 2.13.0 amd64 [installed,upgradable to: 2.14.0]
3. What steps have you already taken to try and solve the problem?
Tried
Looking at config, all looks correct:
secdoc@cerebro:~$ sudo cat /etc/opensearch/opensearch.yml
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.type: single-node
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Remote Store -----------------------------------
# Controls whether cluster imposes index creation only with remote store enabled
# cluster.remote_store.enabled: true
#
# Repository to use for segment upload while enforcing remote store for an index
# cluster.remote_store.repository: my-repo-1
#
# Controls whether cluster imposes index creation only with translog remote store enabled
# cluster.remote_store.translog.enabled: true
#
# Repository to use for translog upload while enforcing remote store for an index
# cluster.remote_store.translog.repository: my-repo-1
#
# ---------------------------------- Experimental Features -----------------------------------
#
# Gates the visibility of the index setting that allows changing of replication type.
# Once the feature is ready for production release, this feature flag can be removed.
#
#opensearch.experimental.feature.replication_type.enabled: false
#
#
# Gates the visibility of the index setting that allows persisting data to remote store along with local disk.
# Once the feature is ready for production release, this feature flag can be removed.
#
#opensearch.experimental.feature.remote_store.enabled: false
#
#
# Gates the functionality of a new parameter to the snapshot restore API
# that allows for creation of a new index type that searches a snapshot
# directly in a remote repository without restoring all index data to disk
# ahead of time.
#
#opensearch.experimental.feature.searchable_snapshot.enabled: false
#
#
# Gates the functionality of enabling extensions to work with OpenSearch.
# This feature enables applications to extend features of OpenSearch outside of
# the core.
#
#opensearch.experimental.feature.extensions.enabled: false
#
#
# Gates the search pipeline feature. This feature enables configurable processors
# for search requests and search responses, similar to ingest pipelines.
#opensearch.experimental.feature.search_pipeline.enabled: false
######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
action.auto_create_index: false
plugins.security.disabled: true
The server log is showing the following:
secdoc@cerebro:~$ tail -f /var/log/graylog-server/server.log
2024-05-25T23:41:57.704Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:41:57.705Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4014
2024-05-25T23:42:02.706Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:42:02.706Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4015
2024-05-25T23:42:07.707Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:42:07.707Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4016
2024-05-25T23:42:12.708Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:42:12.708Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4017
2024-05-25T23:42:17.709Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:42:17.709Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4018
2024-05-25T23:42:22.710Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2024-05-25T23:42:22.710Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #4019
secdoc@cerebro:~$ sudo journalctl -xeu opensearch.service
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit opensearch.service completed and consumed the indicated resources.
May 25 23:57:55 cerebro systemd[1]: Starting OpenSearch...
░░ Subject: A start job for unit opensearch.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit opensearch.service has begun execution.
░░
░░ The job identifier is 25166.
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: A terminally deprecated method in java.lang.System has been called
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:>
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: System::setSecurityManager will be removed in a future release
May 25 23:57:56 cerebro systemd-entrypoint[207573]: May 25, 2024 11:57:56 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: COMPAT locale provider will be removed in a future release
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: A terminally deprecated method in java.lang.System has been called
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/u>
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
May 25 23:57:56 cerebro systemd-entrypoint[207573]: WARNING: System::setSecurityManager will be removed in a future release
May 25 23:57:58 cerebro systemd-entrypoint[207573]: uncaught exception in thread [main]
May 25 23:57:58 cerebro systemd-entrypoint[207573]: java.lang.IllegalStateException: cannot downgrade a node from version [2.14.0] to version [2.13.0]
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:107)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:502)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:394)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:301)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.node.Node.<init>(Node.java:535)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.node.Node.<init>(Node.java:417)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.cli.Command.main(Command.java:101)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
May 25 23:57:58 cerebro systemd-entrypoint[207573]: For complete error details, refer to the log at /var/log/opensearch/graylog.log
May 25 23:59:10 cerebro systemd[1]: opensearch.service: start operation timed out. Terminating.
May 25 23:59:10 cerebro systemd[1]: opensearch.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit opensearch.service has entered the 'failed' state with result 'timeout'.
May 25 23:59:10 cerebro systemd[1]: Failed to start OpenSearch.
░░ Subject: A start job for unit opensearch.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit opensearch.service has finished with a failure.
░░
░░ The job identifier is 25166 and the job result is failed.
May 25 23:59:10 cerebro systemd[1]: opensearch.service: Consumed 18.927s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit opensearch.service completed and consumed the indicated resources.
4. How can the community help?
Looking for possible solution directions…
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]