Graylog will not start after upgrade fron 5.1 to 5.2 - Opensearch Fails to Start

Graylog Central (peer support)
, , , ,
1

1. Describe your incident:

Previously running Graylog Open 5.1. I upgraded to 5.2 following the docs. After doing the obligatory reboot, thew WebUI no longer pulls up now getting the following errors

2. Describe your environment:

  • OS Information:
OS: Ubuntu 22.04.3 LTS x86_64 
Kernel: 5.15.0-88-generic 
Shell: bash 5.1.16 
CPU: AMD Ryzen 9 6900HX with Radeon Graphics (16) @ 3.300GHz
  • Package Version:
secdoc@cerebro:~$ sudo apt list --installed | grep 'mongo\|elasticsearch\|opensearch\|graylog'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

graylog-5.2-repository/stable,now 1-2 all [installed]
graylog-server/stable,now 5.2.0-7 amd64 [installed]
libmongoc-1.0-0/jammy,now 1.21.0-1build1 amd64 [installed,automatic]
libmongocrypt0/jammy,now 1.3.0-1ubuntu1 amd64 [installed,automatic]
mongodb-database-tools/jammy,now 100.9.1 amd64 [installed,automatic]
mongodb-mongosh/jammy,now 1.9.0 amd64 [installed,upgradable to: 2.0.2]
mongodb-org-database-tools-extra/jammy,now 6.0.11 amd64 [installed,automatic]
mongodb-org-database/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-mongos/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-server/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-shell/jammy,now 6.0.11 amd64 [installed,automatic]
mongodb-org-tools/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
opensearch/stable,now 2.5.0 amd64 [installed,upgradable to: 2.11.0]
  • Service logs, configurations, and environment variables:

If I do a tail -f /var/log/graylog-server/server.log I get the following:

ecdoc@cerebro:~$ tail -f /var/log/graylog-server/server.log
2secdoc@cerebro:~$ tail -f /var/log/graylog-server/server.log
com.github.joschi.jadconfig.ValidationException: Parameter password_secret should not be blank
	at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:25) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:11) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:227) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:143) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:101) ~[graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:498) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:286) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:260) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]
2023-11-09T22:12:53.964Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[frontend_hotkeys=on, field_types_management=on, cloud_inputs=on, scripting_api_preview=on, composable_index_templates=off, search_filter=on, preflight_web=on, instant_archiving=off]}
2023-11-09T22:12:54.067Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Parameter password_secret should not be blank
	at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:25) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:11) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:227) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:143) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:101) ~[graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:498) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:286) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:260) [graylog.jar:?]

I restored the server.conf file from backup and get the following errors now:

secdoc@cerebro:~$ sudo tail -f /var/log/graylog-server/server.log
2023-11-10T00:46:43.479Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:46:43.480Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #8
2023-11-10T00:46:48.482Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:46:48.483Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #9
2023-11-10T00:46:53.486Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:46:53.487Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #10
2023-11-10T00:46:58.489Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:46:58.490Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #11
2023-11-10T00:47:03.492Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:03.493Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #12
2023-11-10T00:47:08.496Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:08.496Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #13
2023-11-10T00:47:13.499Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:13.500Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #14
2023-11-10T00:47:18.503Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:18.504Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #15
2023-11-10T00:47:23.507Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:23.508Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #16
2023-11-10T00:47:28.510Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:28.511Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #17
2023-11-10T00:47:33.514Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:33.515Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #18
2023-11-10T00:47:38.517Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:38.518Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #19
2023-11-10T00:47:43.521Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:43.522Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #20
2023-11-10T00:47:48.524Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-11-10T00:47:48.525Z INFO  [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #21

Also running netstat -ntlp shows the following:

secdoc@cerebro:~$ netstat -ntlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:42255         0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8125          0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:19999           0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::19999                :::*                    LISTEN      -

Looking at opensearch service:

[sudo] password for secdoc: 
× opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Fri 2023-11-10 01:11:47 UTC; 21s ago
       Docs: https://opensearch.org/
    Process: 886 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status=143)
   Main PID: 886 (code=exited, status=143)
        CPU: 13.288s

Nov 10 01:10:38 cerebro systemd-entrypoint[886]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Nov 10 01:10:38 cerebro systemd-entrypoint[886]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Nov 10 01:10:38 cerebro systemd-entrypoint[886]:         at org.opensearch.cli.Command.main(Command.java:101)
Nov 10 01:10:38 cerebro systemd-entrypoint[886]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Nov 10 01:10:38 cerebro systemd-entrypoint[886]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Nov 10 01:10:38 cerebro systemd-entrypoint[886]: For complete error details, refer to the log at /var/log/opensearch/graylog.log
Nov 10 01:11:47 cerebro systemd[1]: opensearch.service: start operation timed out. Terminating.
Nov 10 01:11:47 cerebro systemd[1]: opensearch.service: Failed with result 'timeout'.
Nov 10 01:11:47 cerebro systemd[1]: Failed to start OpenSearch.
Nov 10 01:11:47 cerebro systemd[1]: opensearch.service: Consumed 13.288s CPU time.

and the referenced log /var/log/opensearch/graylog.log shows the following:

[2023-11-10T01:02:15,159][INFO ][o.o.n.Node               ] [cerebro] version[2.5.0], pid[136089], build[deb/b8a8b6c4d7fc7a7e32eb2cb68ecad8057a4636ad/2023-01-18T23:48:43.426713304Z], OS[Linux/5.15.0-88-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.5/17.0.5+8]
[2023-11-10T01:02:15,172][INFO ][o.o.n.Node               ] [cerebro] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-11-10T01:02:15,173][INFO ][o.o.n.Node               ] [cerebro] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-6380873967750578433, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2023-11-10T01:02:15,705][WARN ][stderr                   ] [cerebro] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-11-10T01:02:15,705][WARN ][stderr                   ] [cerebro] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-11-10T01:02:15,705][WARN ][stderr                   ] [cerebro] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-11-10T01:02:15,712][INFO ][o.o.s.s.t.SSLConfig      ] [cerebro] SSL dual mode is disabled
[2023-11-10T01:02:15,713][WARN ][o.o.s.OpenSearchSecurityPlugin] [cerebro] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
[2023-11-10T01:02:15,976][INFO ][o.o.p.c.PluginSettings   ] [cerebro] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-11-10T01:02:16,239][INFO ][o.o.i.r.ReindexPlugin    ] [cerebro] ReindexPlugin reloadSPI called
[2023-11-10T01:02:16,240][INFO ][o.o.i.r.ReindexPlugin    ] [cerebro] Unable to find any implementation for RemoteReindexExtension
[2023-11-10T01:02:16,258][INFO ][o.o.j.JobSchedulerPlugin ] [cerebro] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-11-10T01:02:16,271][INFO ][o.o.j.JobSchedulerPlugin ] [cerebro] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-11-10T01:02:16,272][INFO ][o.o.j.JobSchedulerPlugin ] [cerebro] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-11-10T01:02:16,284][INFO ][o.o.j.JobSchedulerPlugin ] [cerebro] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [aggs-matrix-stats]
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [analysis-common]
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [geo]
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [ingest-common]
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [ingest-geoip]
[2023-11-10T01:02:16,289][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [ingest-user-agent]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [lang-expression]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [lang-mustache]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [lang-painless]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [mapper-extras]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [opensearch-dashboards]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [parent-join]
[2023-11-10T01:02:16,290][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [percolator]
[2023-11-10T01:02:16,291][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [rank-eval]
[2023-11-10T01:02:16,291][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [reindex]
[2023-11-10T01:02:16,291][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [repository-url]
[2023-11-10T01:02:16,291][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [systemd]
[2023-11-10T01:02:16,291][INFO ][o.o.p.PluginsService     ] [cerebro] loaded module [transport-netty4]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-alerting]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-anomaly-detection]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-asynchronous-search]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-cross-cluster-replication]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-geospatial]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-index-management]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-job-scheduler]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-knn]
[2023-11-10T01:02:16,292][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-ml]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-neural-search]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-notifications]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-notifications-core]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-observability]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-performance-analyzer]
[2023-11-10T01:02:16,293][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-reports-scheduler]
[2023-11-10T01:02:16,294][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-security]
[2023-11-10T01:02:16,294][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-security-analytics]
[2023-11-10T01:02:16,294][INFO ][o.o.p.PluginsService     ] [cerebro] loaded plugin [opensearch-sql]
[2023-11-10T01:02:16,333][INFO ][o.o.e.NodeEnvironment    ] [cerebro] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [1.5tb], net total_space [1.8tb], types [ext4]
[2023-11-10T01:02:16,334][INFO ][o.o.e.NodeEnvironment    ] [cerebro] heap size [1gb], compressed ordinary object pointers [true]
[2023-11-10T01:02:16,408][ERROR][o.o.b.Bootstrap          ] [cerebro] Exception
java.lang.IllegalArgumentException: Could not load codec 'Lucene95'. Did you forget to add lucene-backward-codecs.jar?
        at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:515) ~[lucene-core-9.4.2.jar:9.4.2 858d9b437047a577fa9457089afff43eefa461db - jpountz - 2022-11-17 12:56:39]
        at org.apache.lucene.index.SegmentInfos.parseSegmentInfos(SegmentInfos.java:404) ~[lucene-core-9.4.2.jar:9.4.2 858d9b437047a577fa9457089afff43eefa461db - jpountz - 2022-11-17 12:56:39]
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:363) ~[lucene-core-9.4.2.jar:9.4.2 858d9b437047a577fa9457089afff43eefa461db - jpountz - 2022-11-17 12:56:39]
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:299) ~[lucene-core-9.4.2.jar:9.4.2 858d9b437047a577fa9457089afff43eefa461db - jpountz - 2022-11-17 12:56:39]

Running sudo journalctl -xeu opensearch.service shows the following:

A start job for unit opensearch.service has finished with a failure.
░░ 
░░ The job identifier is 164 and the job result is failed.
Nov 10 01:11:47 cerebro systemd[1]: opensearch.service: Consumed 13.288s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit opensearch.service completed and consumed the indicated resources.
Nov 10 01:20:43 cerebro systemd[1]: Starting OpenSearch...
░░ Subject: A start job for unit opensearch.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit opensearch.service has begun execution.
░░ 
░░ The job identifier is 2549.
Nov 10 01:20:44 cerebro systemd-entrypoint[12298]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 10 01:20:44 cerebro systemd-entrypoint[12298]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.5.0.jar)
Nov 10 01:20:44 cerebro systemd-entrypoint[12298]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 10 01:20:44 cerebro systemd-entrypoint[12298]: WARNING: System::setSecurityManager will be removed in a future release
Nov 10 01:20:45 cerebro systemd-entrypoint[12298]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 10 01:20:45 cerebro systemd-entrypoint[12298]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.5.0.jar)
Nov 10 01:20:45 cerebro systemd-entrypoint[12298]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 10 01:20:45 cerebro systemd-entrypoint[12298]: WARNING: System::setSecurityManager will be removed in a future release
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]: uncaught exception in thread [main]
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]: java.lang.IllegalArgumentException: Could not load codec 'Lucene95'. Did you forget to add lucene-backward-codecs.jar?
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]: Likely root cause: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene95' does not exist.  You need to add the corresponding JAR file supporting this SPI to your cl>
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.codecs.Codec.forName(Codec.java:118)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:511)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.SegmentInfos.parseSegmentInfos(SegmentInfos.java:404)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:363)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:299)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:88)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:809)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:309)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:453)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:369)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.node.Node.<init>(Node.java:456)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.node.Node.<init>(Node.java:356)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.cli.Command.main(Command.java:101)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Nov 10 01:20:46 cerebro systemd-entrypoint[12298]: For complete error details, refer to the log at /var/log/opensearch/graylog.log
Nov 10 01:21:58 cerebro systemd[1]: opensearch.service: start operation timed out. Terminating.
Nov 10 01:21:58 cerebro systemd[1]: opensearch.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit opensearch.service has entered the 'failed' state with result 'timeout'.
Nov 10 01:21:58 cerebro systemd[1]: Failed to start OpenSearch.
░░ Subject: A start job for unit opensearch.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit opensearch.service has finished with a failure.
░░ 
░░ The job identifier is 2549 and the job result is failed.
Nov 10 01:21:58 cerebro systemd[1]: opensearch.service: Consumed 12.167s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit opensearch.service completed and consumed the indicated resources.

3. What steps have you already taken to try and solve the problem?

I have tried starting and stopping the services and rebooting. Trying to look at the log just show the java error listed above. After double checking the server.conf file, I found that it had been overwritten and restored from a backup. and that is when I started getting the other errors in the log.

4. How can the community help?
I am looking for assistance on the direction to resolve the issue.This is the first time I have had an issue with the update and cannot seem to find any reference in other posts to this particular log error,

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @secdoc

After looking over you logs the first one you posted

Graylog configuration file should of had that set already as shown below from the documentation.

I quote:

Additionally add password_secret and root_password_sha2 as these are mandatory and Graylog will not start without them.

Found here

Opensearch failed to start.

What version of OpenSearch are you using? And how did you install opensearch?
Its not clear how you executed this upgrade. BTW Linux is awesome because you dont need to reboot your server unles its a kernel update :+1:

Can you show the opensearch configuration file here? If you can just make sure you hide personal info.

After reading this post It seems to me, and I believe you stated, it wrote over the existing configuration files? The first issue clued me in on that because version 5.1 would not be working if that configuration was absent.

I personal would look over Opensearch configuration file, and double check the configs.

If need be, check Opensearch version compatibility with Graylog.
Check Permissions just incase. If you set opensearch repo in Ubuntu you could execute a reinstall. just an idea.

 root # sudo apt reinstall opensearch.

Ubuntu might ask to keep the old files or write over them. Press “n” and keep the old configuration file. This all depends on your version and/or how you install opensearch.

Have you tried to execute a full upgrade on this node, see if that works?

root # apt update && apt upgrade

EDIT:
Just incase, this should probably be the only configurations in opensearch.yml file and the rest should be commented out…

cluster.name: graylog
node.name: My_host
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
discovery.type: single-node
network.host: 0.0.0.0
action.auto_create_index: false
plugins.security.disabled: true
1 Like
3

@gsmith -

Thank you for the reply. You are correct that the server.conf file was not initial configured. The original had been overwritten this time during the update, but if you notice further on in my post, I updated from the backup.

As far as the version of opensearch, it is listed in the package list but it is: opensearch/stable,now 2.5.0 amd64 [installed,upgradable to: 2.11.0]
I have been running graylog since version 5.0 was released without issue. Opensearch was installed using documentation at Graylog Install Page and also referencing OpenSearch Install Page.

Here is the Opensearch .yml configuration file (just to note because I neglected to list it earlier, I did uncomment the for the 9200 listening port which was done in troubleshooting):

# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.type: single-node
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Remote Store -----------------------------------
# Controls whether cluster imposes index creation only with remote store enabled
# cluster.remote_store.enabled: true
#
# Repository to use for segment upload while enforcing remote store for an index
# cluster.remote_store.repository: my-repo-1
#
# Controls whether cluster imposes index creation only with translog remote store enabled
# cluster.remote_store.translog.enabled: true
#
# Repository to use for translog upload while enforcing remote store for an index
# cluster.remote_store.translog.repository: my-repo-1
#
# ---------------------------------- Experimental Features -----------------------------------
#
# Gates the visibility of the index setting that allows changing of replication type.
# Once the feature is ready for production release, this feature flag can be removed.
#
#opensearch.experimental.feature.replication_type.enabled: false
#
#
# Gates the visibility of the index setting that allows persisting data to remote store along with local disk.
# Once the feature is ready for production release, this feature flag can be removed.
#
#opensearch.experimental.feature.remote_store.enabled: false
#
#
# Gates the functionality of a new parameter to the snapshot restore API
# that allows for creation of a new index type that searches a snapshot
# directly in a remote repository without restoring all index data to disk
# ahead of time.
#
#opensearch.experimental.feature.searchable_snapshot.enabled: false
#
#
#
#
# Gates the search pipeline feature. This feature enables configurable processors
# for search requests and search responses, similar to ingest pipelines.
#opensearch.experimental.feature.search_pipeline.enabled: false

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-det>
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
action.auto_create_index: false
plugins.security.disabled: true

You are correct on the reboot aspect. Unfortunately, a reboot was needed for for another reason and that is what triggered the issue.prior to the reboot, the box had been running for almost a year.

4

@gsmith thank you for the tip. The reinstall of opensearch seemed to clear the issue. Not sure what the hangup was specifically the reinstall did the trick. I did not even think about a reinstall as an option…

The reinstall moved opensearch from 2.5.0 to 2.11.0:

secdoc@cerebro:~$ sudo apt list --installed | grep 'mongo\|elasticsearch\|opensearch\|graylog'
[sudo] password for secdoc: 

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

graylog-5.2-repository/stable,now 1-2 all [installed]
graylog-server/stable,now 5.2.0-7 amd64 [installed]
libmongoc-1.0-0/jammy,now 1.21.0-1build1 amd64 [installed,automatic]
libmongocrypt0/jammy,now 1.3.0-1ubuntu1 amd64 [installed,automatic]
mongodb-database-tools/jammy,now 100.9.1 amd64 [installed,automatic]
mongodb-mongosh/jammy,now 1.9.0 amd64 [installed,upgradable to: 2.0.2]
mongodb-org-database-tools-extra/jammy,now 6.0.11 amd64 [installed,automatic]
mongodb-org-database/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-mongos/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-server/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org-shell/jammy,now 6.0.11 amd64 [installed,automatic]
mongodb-org-tools/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
mongodb-org/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.11]
opensearch/stable,now 2.11.0 amd64 [installed]

Thank you again.

1 Like
5

Hey @secdoc

In the future you should pin you opensearch repo version.

https://help.ubuntu.com/community/PinningHowto

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.