Graylog server is not starting

I have installed step to step from documentation but graylog server is not starting. Mongodb server is also running properly

graylog/server.log
> ubuntu@ip-172-31-42-22:~$ tail -f /var/log/graylog-server/server.log

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[graylog.jar:?]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[graylog.jar:?]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[graylog.jar:?]
… 3 more
2021-04-29T05:48:53.286Z INFO [connection] Opened connection [connectionId{localValue:15, serverValue:1}] to localhost:27017
2021-04-29T05:48:53.287Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 24]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=306404}
2021-04-29T05:51:21.452Z INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.6 [org.graylog.aws.AWSPlugin]
2021-04-29T05:51:21.457Z INFO [CmdLineTool] Loaded plugin: Collector 4.0.6 [org.graylog.plugins.collector.CollectorPlugin]
2021-04-29T05:51:21.458Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-04-29T05:51:21.458Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-04-29T05:51:21.460Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-04-29T05:51:21.917Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-04-29T05:51:22.435Z INFO [Version] HV000001: Hibernate Validator null
2021-04-29T05:51:28.764Z INFO [InputBufferImpl] Message journal is enabled.
2021-04-29T05:51:28.793Z INFO [NodeId] Node ID: a55e7341-b246-45e0-9aa9-963b7832aaca
2021-04-29T05:51:29.285Z INFO [LogManager] Loading logs.
2021-04-29T05:51:29.321Z WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index…
2021-04-29T05:51:29.362Z INFO [LogManager] Logs loading complete.
2021-04-29T05:51:29.368Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-04-29T05:51:29.415Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-04-29T05:51:29.487Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-04-29T05:51:29.542Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:3}] to localhost:27017
2021-04-29T05:51:29.551Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 24]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=7460950}
2021-04-29T05:51:29.577Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:4}] to localhost:27017
2021-04-29T05:51:29.970Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.

@kesifalaraskey
Hello,
Is graylog service not starting or are you unable to log into Graylog web interface?
Those logs dont really show me what the problem is. Some more details would be helpful.
Thanks

1 Like

Thank you for your quick response.

graylog itself not starting.

ubuntu@ip-172-31-42-22:~$ netstat -nptl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -

1 Like

Ok,
Going to ask you a few question.
Can you execute this command and what does it show you? Do you see any errors?

systemctl status graylog-server

What do you see when you execute this?

curl -XGET ‘http://localhost:9200/_cluster/health?pretty=true

ubuntu@ip-172-31-42-22:~$ systemctl status graylog-server
● graylog-server.service - Graylog server
Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-04-29 04:26:50 UTC; 14min ago
Docs: http://docs.graylog.org/
Main PID: 1168 (graylog-server)
Tasks: 19 (limit: 1160)
Memory: 623.3M
CGroup: /system.slice/graylog-server.service
├─1168 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─1202 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSw>

Apr 29 04:26:50 ip-172-31-42-22 systemd[1]: Stopped Graylog server.
Apr 29 04:26:50 ip-172-31-42-22 systemd[1]: Started Graylog server.

I am using aws elasticsearch and on curl it is responsing

{
“name” : “121…2db”,
“cluster_name” : “2…:p…”,
“cluster_uuid” : “0._…g”,
“version” : {
“number” : “x.xx.x-SNAPSHOT”,
“build_flavor” : “oss”,
“build_type” : “tar”,
“build_hash” : “unknown”,
“build_date” : “2021-03-10T10:58:06.207203Z”,
“build_snapshot” : true,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

@kesifalaraskey

Looks like Graylog service is running. Sorry I’m not familiar with AWS elasticsearch.
Only suggetion I could give you is check you firewall make sure the correct ports are opened and look through MongoDb, Elasticsearch and Graylog server log files. Maybe something in there can tell you the problem. If you could post where you got your instruction from and how you configured you graylog configuration file. Someone here might be able to help you further.
If your unsure what to post take a look at this.

https://community.graylog.org/t/community-guidelines/6649#details

@kesifalaraskey
I just seen the bottom of your post, I’m sorry. Maybe try this instead

curl -XGET ‘http://127.0.0.1:9200/_cluster/health?pretty=true’

You should see something like this.

Server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =kNImNQozKstySv…r5vP0BI1hHvuTCYDqnptkzU
root_password_sha2 =433fce394f…f40cbb
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address : 127.0.0.1:9000(default)
elasticsearch_hosts = ElasticEndpoint

mostly all thinks are default

{
“cluster_name”:“2…7:p…”,
“status”:“yellow”,
“timed_out”:false,
“number_of_nodes”:1,
“number_of_data_nodes”:1,
“discovered_master”:true,
“active_primary_shards”:12,
“active_shards”:12,
“relocating_shards”:0,
“initializing_shards”:0,
“unassigned_shards”:10,
“delayed_unassigned_shards”:0,
“number_of_pending_tasks”:0,
“number_of_in_flight_fetch”:0,
“task_max_waiting_in_queue_millis”:0,
“active_shards_percent_as_number”:54.54545454545454
}

Get yellow status and also if elastic search have problem than also server can start ?

Even thou its yellow , you should be able to log into the Graylog’s Web UI.

Try using this and restart graylog service.

http_bind_address: 0.0.0.0:9000

Not sure about that configuration.
should look something like this, which I believe is a default config.

elasticsearch_hosts = http://127.0.0.1:9200

1 Like

Nothing happen same log return as shown in logs it is not connecting with even mongod

Ok ,
Do you have this set in you Graylog config it should be uncommented.

mongodb_uri = mongodb://localhost/graylog

Nothing change happen.

Mongod.logs

2021-04-29T05:24:03.150+0000 I NETWORK [listener] connection accepted from 127.0.0.1:47900 #4 (1 connection now open)
2021-04-29T05:24:03.167+0000 I NETWORK [conn4] received client metadata from 127.0.0.1:47900 conn4: { driver: { name: “mongo-java-driver|legacy”, version: “3.12.1” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “5.4.0-1038-aws” }, platform: “Java/Private Build/1.8.0_282-8u282-b08-0ubuntu1~20.04-b08” }
2021-04-29T05:24:03.229+0000 I NETWORK [listener] connection accepted from 127.0.0.1:47902 #5 (2 connections now open)
2021-04-29T05:24:03.230+0000 I NETWORK [conn5] received client metadata from 127.0.0.1:47902 conn5: { driver: { name: “mongo-java-driver|legacy”, version: “3.12.1” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “5.4.0-1038-aws” }, platform: “Java/Private Build/1.8.0_282-8u282-b08-0ubuntu1~20.04-b08” }
2021-04-29T05:28:42.311+0000 I NETWORK [conn5] end connection 127.0.0.1:47902 (1 connection now open)
2021-04-29T05:28:42.312+0000 I NETWORK [conn4] end connection 127.0.0.1:47900 (0 connections now open)
2021-04-29T05:28:52.814+0000 I NETWORK [listener] connection accepted from 127.0.0.1:47914 #6 (1 connection now open)
2021-04-29T05:28:52.831+0000 I NETWORK [conn6] received client metadata from 127.0.0.1:47914 conn6: { driver: { name: “mongo-java-driver|legacy”, version: “3.12.1” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “5.4.0-1038-aws” }, platform: “Java/Private Build/1.8.0_282-8u282-b08-0ubuntu1~20.04-b08” }
2021-04-29T05:28:52.879+0000 I NETWORK [listener] connection accepted from 127.0.0.1:47916 #7 (2 connections now open)
2021-04-29T05:28:52.881+0000 I NETWORK [conn7] received client metadata from 127.0.0.1:47916 conn7: { driver: { name: “mongo-java-driver|legacy”, version: “3.12.1” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “5.4.0-1038-aws” }, platform: “Java/Private Build/1.8.0_282-8u282-b08-0ubuntu1~20.04-b08” }

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.