FAIL TO START - elasticsearch

adrianrus · September 20, 2022, 3:37pm

I just installed Garylog on a new VM but i get this issue.

root~# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-09-20 15:04:56 UTC; 6s ago
       Docs: https://www.elastic.co
    Process: 3414 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 3414 (code=exited, status=1/FAILURE)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 20 15:04:56 ipt-fr-log-01 systemd-entrypoint[3414]:         ... 9 more
Sep 20 15:04:56 ipt-fr-log-01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 15:04:56 ipt-fr-log-01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 20 15:04:56 ipt-fr-log-01 systemd[1]: Failed to start Elasticsearch.

Content of elasticsearch.yml

# ======================== Elasticsearch Configuration =========================#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog
action.auto_create_index: false
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["127.0.0.1", "[::1]"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: node-1
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
cluster.name: graylog
action.auto_create_index: false

after running journalctl -u elasticsearch

-- Logs begin at Sat 2022-09-17 18:14:59 UTC, end at Tue 2022-09-20 15:33:37 UTC. --
Sep 20 10:16:10 -01 systemd[1]: Starting Elasticsearch...
Sep 20 10:16:12 -01 systemd-entrypoint[26209]: Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonPar>
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]];
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 20 10:16:12 -01 systemd-entrypoint[26209]:         ... 10 more
Sep 20 10:16:13 -01 systemd-entrypoint[26126]: Exception in thread "main" 2022-09-20 10:16:13,963 main ERROR No Log4j 2 configuration file found. Using defaul>
Sep 20 10:16:13 -01 systemd-entrypoint[26126]: SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplicate field>
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]];
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 20 10:16:13 -01 systemd-entrypoint[26126]:         ... 9 more
Sep 20 10:16:14 -01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 10:16:14 -01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

I tried several variants of elasticsearch.YML variables, still nothing.

As a consequence:

root:~# curl -X GET http://localhost:9200
curl: (7) Failed to connect to localhost port 9200: Connection refused

tmacgbay · September 20, 2022, 7:22pm

You have cluster.name twice in the elasticsearch.yml file - yml files are very exacting with spacing and indentation as well.

If this is a new installation and you are using 4.3.x and above, you should use OpenSearch instead of Elasticsearch. If you are less than Graylog 4.3 for some reason or you want to stay with Elasticsearch (for some reason) make sure you only go as high as Elasticsearch 7.10.x … anything newer has compatibility and licensing issues.

adrianrus · September 21, 2022, 7:28am

Hi,
I’ll go for the Open Search, in this case, as i got 4.3.1 Graylog, therefore will open another thread, as that one is not working as well

adrianrus · September 21, 2022, 8:10am

Still, i am trying to understand you guys.

Why do you propose to install ES in your guide, and then, you say ES is not supported on the newer 4.3.x :)) - am i missing something?

https://docs.graylog.org/docs/ubuntu

I followed all the steps precisely, and got that issue with ES not starting.
Then i started to dig for troubleshoot and i ended up with that duplicate, which, still, was not the cause of the problem.

Regarding the Open Source, there is even less comprehensive info - nothing on ubuntu, still, i’m trying to make it work :))

tmacgbay · September 21, 2022, 1:13pm

Docs are a little behind. They were converted to a new platform a while back and the results are a little quirky. Add in that the jump over to OpenSearch has wasn’t originally planned at the same time… but that was a while ago… come on Doc Dudes, get movin!!! Basically Graylog is caught in a transitional period.

OpenSearch and Ubuntu is still a tarball to work with… not optimal… but they are working on an apt install… Graylog is helping them on that.

Chances are that Elastic is not starting for a different reason now, post up the journalctl and lets look at it… (Side note - some people are having issues getting OpenSearch to run too… I still need to get that going myself.)

adrianrus · September 21, 2022, 2:05pm

if i tell you how i made it working you will die laughing, but it did the trick.

Si started froma fresh VM, Ubuntu 20.0
Installed GL 3.3 with Mongo 4.0 and ES 6.8

After got it working, snapshot
Then, upgraded GL to 4.0
then ES to 7.1

After got it working took snapshot.

Now i am on the next step for GL 4.3 upgrade.
Maybe i’ll make a YT video with his.

tmacgbay · September 21, 2022, 3:06pm

system · October 5, 2022, 3:07pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Starting Elasticsearch Service fail after upgrade Graylog Central (peer support)	7	3042	May 11, 2022
Elasticsearch Service Not Running (New Installation) Graylog Central (peer support)	7	7211	October 9, 2018
Graylog installation manual for ubuntu improvements - elasticsearch issue Graylog Central (peer support)	8	1784	May 24, 2019
Elasticsearch service failed(2.5->6.8 upgrade) Graylog Central (peer support)	2	590	December 12, 2019
ElasticSearch wont stay running Graylog Central (peer support)	8	1184	November 4, 2020

FAIL TO START - elasticsearch

Related topics