someshow elasticsearch did not finish correctly and now .lock files keep the service from starting (edit: at least such was my initial impression)
My approach would be to remove all .lock files for elasticsearch but i’m afraid this may destroy all data, is this correct ? I’d rather loose some data than all data
many entries for /var/lib/elasticsearch/nodes/0/indices/…/?/write.lock
and a single one for /var/lib/elasticsearch/nodex/0/node.lock
Reason for my asking is i could not find relevant documentation which explains why and what adequately. Pointers are most welcome.
found nothing useful really, almost all i found resemble this output, maybe i’m missing something
Nov 30 18:48:36 graylog systemd[1]: Started Elasticsearch.
Nov 30 18:48:36 graylog kernel: audit: type=1130 audit(1606758516.631:4314): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg=‘unit=elasticsearch comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success’
Nov 30 18:48:36 graylog elasticsearch[32007]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 30 18:48:40 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 18:48:40 graylog systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.
Nov 30 18:48:40 graylog kernel: audit: type=1131 audit(1606758520.247:4315): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg=‘unit=elasticsearch comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed’
---- MORE output was found after removing --quiet in systemd startup
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.node.Node.(Node.java:339) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: … 6 more
Nov 30 19:00:43 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 19:00:43 graylog systemd[1]: elasticsearch.service: Failed with result ‘exit-code’
AH, finally… i cannot believe it a dependency on a minor version difference !
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,018][INFO ][o.e.e.NodeEnvironment ] [graylog-gray] using [1] data paths, mounts [[/var (/dev/mapper/graylog–vg-var)]], net usable_space [12.7gb]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,021][INFO ][o.e.e.NodeEnvironment ] [graylog-gray] heap size [1.9gb], compressed ordinary object pointers [true]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,536][INFO ][o.e.n.Node ] [graylog-gray] node name [graylog-gray], node ID [Nv8T_4gPTma_sxemhWkzUw]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,537][INFO ][o.e.n.Node ] [graylog-gray] version[6.8.13], pid[2173], build[oss/deb/be13c69/2020-10-16T09:09:46.555371Z], OS[Linux/4.19
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,537][INFO ][o.e.n.Node ] [graylog-gray] JVM arguments [-Xms1g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75
Nov 30 19:29:08 graylog elasticsearch[2173]: [2020-11-30T19:29:08,272][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [graylog-gray] uncaught exception in thread [main] Nov 30 19:29:08 graylog elasticsearch[2173]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [analysis-icu] was built for Elasticsearch version 6.8.12 but version 6.8.13 i
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.13.jar:6.8.13] Nov 30 19:29:08 graylog elasticsearch[2173]: Caused by: java.lang.IllegalArgumentException: Plugin [analysis-icu] was built for Elasticsearch version 6.8.12 but version 6.8.13 is running
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:346) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:531) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.node.Node.(Node.java:339) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: … 6 more
Nov 30 19:29:08 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE.
should tell you if a plugin is relevant, if ONE is, ALL plugins are likely relevant so upgrade them ALL (unless you know better and it actually could destory your data, obviously)
esimpleupgrade.sh
#!/bin/bash
upl=`ls -1 /usr/share/elasticsearch/plugins/`
for i in $upl
do
/usr/share/elasticsearch/bin/elasticsearch-plugin remove $i
/usr/share/elasticsearch/bin/elasticsearch-plugin install $i
done