Elasticsearch fails to start because of .lock files?

someshow elasticsearch did not finish correctly and now .lock files keep the service from starting (edit: at least such was my initial impression)

My approach would be to remove all .lock files for elasticsearch but i’m afraid this may destroy all data, is this correct ? I’d rather loose some data than all data

many entries for /var/lib/elasticsearch/nodes/0/indices/…/?/write.lock
and a single one for /var/lib/elasticsearch/nodex/0/node.lock

Reason for my asking is i could not find relevant documentation which explains why and what adequately. Pointers are most welcome.

After reading https://lucene.apache.org/core/5_1_0/core/org/apache/lucene/codecs/lucene50/package-summary.html#Lock_File found on https://www.elastic.co/blog/found-dive-into-elasticsearch-storage#lucene-index-files i decided to remove the .lock files in /var/lib/elasticsearch

no change, elasticsearch still refuses to start

:wave: what do your elastic logs say? Or even some systemctl output?

found nothing useful really, almost all i found resemble this output, maybe i’m missing something

Nov 30 18:48:36 graylog systemd[1]: Started Elasticsearch.
Nov 30 18:48:36 graylog kernel: audit: type=1130 audit(1606758516.631:4314): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg=‘unit=elasticsearch comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success’
Nov 30 18:48:36 graylog elasticsearch[32007]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 30 18:48:40 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 18:48:40 graylog systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.
Nov 30 18:48:40 graylog kernel: audit: type=1131 audit(1606758520.247:4315): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg=‘unit=elasticsearch comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed’

---- MORE output was found after removing --quiet in systemd startup

Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.node.Node.(Node.java:339) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:00:43 graylog elasticsearch[1854]: … 6 more
Nov 30 19:00:43 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 19:00:43 graylog systemd[1]: elasticsearch.service: Failed with result ‘exit-code’

AH, finally… i cannot believe it a dependency on a minor version difference !

Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,018][INFO ][o.e.e.NodeEnvironment ] [graylog-gray] using [1] data paths, mounts [[/var (/dev/mapper/graylog–vg-var)]], net usable_space [12.7gb]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,021][INFO ][o.e.e.NodeEnvironment ] [graylog-gray] heap size [1.9gb], compressed ordinary object pointers [true]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,536][INFO ][o.e.n.Node ] [graylog-gray] node name [graylog-gray], node ID [Nv8T_4gPTma_sxemhWkzUw]
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,537][INFO ][o.e.n.Node ] [graylog-gray] version[6.8.13], pid[2173], build[oss/deb/be13c69/2020-10-16T09:09:46.555371Z], OS[Linux/4.19
Nov 30 19:29:07 graylog elasticsearch[2173]: [2020-11-30T19:29:07,537][INFO ][o.e.n.Node ] [graylog-gray] JVM arguments [-Xms1g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75
Nov 30 19:29:08 graylog elasticsearch[2173]: [2020-11-30T19:29:08,272][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [graylog-gray] uncaught exception in thread [main]
Nov 30 19:29:08 graylog elasticsearch[2173]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [analysis-icu] was built for Elasticsearch version 6.8.12 but version 6.8.13 i
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: Caused by: java.lang.IllegalArgumentException: Plugin [analysis-icu] was built for Elasticsearch version 6.8.12 but version 6.8.13 is running
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:346) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:531) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.node.Node.(Node.java:339) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.13.jar:6.8.13]
Nov 30 19:29:08 graylog elasticsearch[2173]: … 6 more
Nov 30 19:29:08 graylog systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE.

Fixed by removing and installing elasticsearch plugins, yes this is required for some reason.

copy and comment out --quiet in /usr/lib/systemd/system/elasticsearch.service
such as

#ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p {PID_DIR}/elasticsearch.pid --quiet ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p {PID_DIR}/elasticsearch.pid

save and close the file, now reload systemd stuff and start elastic while watching for pesky plugins not having been upgraded

systemctl daemon-reload
systemctl start elasticsearch && tail -n 500 /var/log/syslog | grep Caused

should tell you if a plugin is relevant, if ONE is, ALL plugins are likely relevant so upgrade them ALL (unless you know better and it actually could destory your data, obviously)

esimpleupgrade.sh

#!/bin/bash
upl=`ls -1 /usr/share/elasticsearch/plugins/`
for i in $upl
do 
 /usr/share/elasticsearch/bin/elasticsearch-plugin remove $i
 /usr/share/elasticsearch/bin/elasticsearch-plugin install $i
done

Oof. Which log was that in? The Elasticsearch log? Does upgrading it allow ES to start?

eventually syslog, had to disable --quiet in the systemctl script to get sensible feedback

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.