Graylog Installation Issue

tmacgbay · October 6, 2022, 11:55pm

Your Elasticsearch service isn’t working - Post logs from Graylog as @gsmith suggests as well as results of:

$ journalctl -eu elasticsearch

Which will show more detail on why Elasticsearch is failing. You can also post results of:

$ cat /etc/elasticsearch/elasticsearch.yml    | egrep -v "^\s*(#|$)"

Which shows your ElasticSearch settings.

emgonzalez59 · October 7, 2022, 1:17am

ubuntu@ubuntu:~$ journalctl -eu elasticsearch
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 22 21:42:26 ubuntu systemd-entrypoint[1151]:         ... 9 more
Sep 22 21:42:26 ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 21:42:26 ubuntu systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 22 21:42:26 ubuntu systemd[1]: Failed to start Elasticsearch.
-- Reboot --
Sep 22 21:56:50 ubuntu systemd[1]: Starting Elasticsearch...
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]: Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplicate field 'cluster.name'
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]];
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 22 21:57:07 ubuntu systemd-entrypoint[1415]:         ... 10 more
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]: Exception in thread "main" 2022-09-22 21:57:10,440 main ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatical>
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]: SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplicate field 'cluster.name'
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]];
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1113)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1086)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 89, column: 13]
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1109)
Sep 22 21:57:10 ubuntu systemd-entrypoint[1151]:         ... 9 more
Sep 22 21:57:10 ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 21:57:10 ubuntu systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 22 21:57:10 ubuntu systemd[1]: Failed to start Elasticsearch.
-- Reboot --
Oct 06 12:42:36 ubuntu systemd[1]: Starting Elasticsearch...
-- Reboot --
Oct 06 12:49:50 ubuntu systemd[1]: Starting Elasticsearch...

ubuntu@ubuntu:~$ cat /etc/elasticsearch/elasticsearch.yml | egrep -v “^\s*(#|$)”
cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: graylog
action.auto_create_index: false

`ubuntu@ubuntu:/usr/local/bin$ tail -f /var/log/graylog-server/server.log
2022-10-06T22:16:16.334-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:16.334-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6792
2022-10-06T22:16:21.335-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:21.335-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6793
2022-10-06T22:16:26.335-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:26.335-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6794
2022-10-06T22:16:31.336-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:31.336-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6795
2022-10-06T22:16:36.336-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:36.337-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6796
2022-10-06T22:16:41.337-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:41.337-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6797
2022-10-06T22:16:46.338-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:46.338-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6798
2022-10-06T22:16:51.339-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:51.339-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6799
2022-10-06T22:16:56.339-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:16:56.339-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6800
2022-10-06T22:17:01.340-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:17:01.340-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6801
2022-10-06T22:17:06.341-03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2022-10-06T22:17:06.341-03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #6802

Thanks in advance,````

gsmith · October 7, 2022, 1:30am

Hello @emgonzalez59
Noticed in your logs it showed this…

Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: JsonParseException[Duplicate field 'cluster.name'

As you shown above…

Something wrong with Elasticsearch yaml file.
It should look something like this default setting

[root@graylog graylog_user]# cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false
discovery.type: single-node
[root@graylog graylog_user]#

EDIT: But this thing is, you should still be able to login Graylog Without Elasticsearch, so I would double check for duplicate configurations, Permissions on files and folders, firewall and/or check Selinux/Apparmor enabled

emgonzalez59 · October 7, 2022, 2:21am

I have deleted the duplicated lines in ElasticSearch config file and corercted the IP address to localhost 127.0.0.1 , and now I can access the Graylog WebUI thru http://IP_Address:9000
FYI here is the output of cat command for elasticsearch,yml file:

ubuntu@ubuntu:/etc/selinux$ cat /etc/elasticsearch/elasticsearch.yml    | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false

On the other hand, I run this command :

ubuntu@ubuntu:/etc/selinux$ curl -X GET http://127.0.0.1:9200
{
“name” : “ubuntu”,
“cluster_name” : “graylog”,
“cluster_uuid” : “VZTpvLCiQZi_gdD03knI6A”,
“version” : {
“number” : “7.10.2”,
“build_flavor” : “oss”,
“build_type” : “deb”,
“build_hash” : “747e1cc71def077253878a59143c1f785afa92b9”,
“build_date” : “2021-01-13T00:42:12.435326Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

What do you think about ElasticSearch? do you think it is running?

Another question (sorry for that…) It is the first time i installed a SIEM like Graylog, I am trying to integrate it with ClearPass Policy Manager NAC. Can I get documentation about it? Thanks again and sorry for this long issue…

gsmith · October 7, 2022, 2:23am

@emgonzalez59 Awesome glad your resolved your issue…

As for

Not sure, are you referring to send logs from that type of device?

emgonzalez59 · October 7, 2022, 2:25am

just to clarify another thing… I have checked that my Ubuntu is NOT running SeLinux (output returns 1) >>
ubuntu@ubuntu:/etc/selinux$ selinuxenabled; echo $?
1

gsmith · October 7, 2022, 2:29am

From what I can see looks like you have everything running.

Also @emgonzalez59
You can use this command to check the status of elasticsearch, If its in “Green” as shown below you should be good

 curl -XGET http://127.0.0.1:9200/_cluster/health?pretty

[root@graylog graylog_user]#  curl -XGET http://10.10.10.10:9200/_cluster/health?pretty
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 436,
  "active_shards" : 436,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@graylog graylog_user]# ^C

emgonzalez59 · October 7, 2022, 12:34pm

hi there,
It seems that ElasticSearch is not ok.

ubuntu@ubuntu:~$ curl -X GET http://192.168.1.2:9200/_cluster/health?pretty
curl: (7) Failed to connect to 192.168.1.2 port 9200: Connection refused

tmacgbay · October 7, 2022, 1:40pm

You were able to see elasticsearch via loopback (127.0.0.1) so put loopback in your cluster check…

ubuntu@ubuntu:~$ curl -X GET http://127.0.0.1:9200/_cluster/health?pretty

emgonzalez59 · October 7, 2022, 6:47pm

Yes, with 127.0.0.1 the output shows Green status.
Then I suppose all is running

Thanks,
Enrique

Obtener Outlook para Android

tmacgbay · October 7, 2022, 7:04pm

Great!! Post any follow on questions as New Topics, we’ll try our best to help!

Topic		Replies	Views
Cannot load graylog web interface from fresh install Graylog Central (peer support)	26	6545	March 22, 2023
Can not Login to Web Gui of latest Graylog Graylog Central (peer support)	18	2472	August 22, 2022
Unable to get graylog started Graylog Central (peer support) basic-configuration	5	3333	February 1, 2022
Loading field information failed with status: cannot GET http://x.x.x.x:9000/api/system/fields (500) Graylog Central (peer support) sidecar , filebeat-windows , winlogbeat	29	8906	November 26, 2018
Graylog Installation JSON problem Graylog Central (peer support)	5	667	July 14, 2021

Graylog Installation Issue

Related topics