Graylog server isn't starting when put AWS Elasticsearch endpoint

[kesifalaraskey (Hafiz Shaikh)]

Try to connet aws elasticsearch with graylog but as soon as i add elasticsearch endpoint graylog server stop working and server.log stop at line

[InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.

graylog server.log

tail -f /var/log/graylog-server/server.log
2021-04-29T12:50:24.224Z INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.6 [org.graylog.aws.AWSPlugin]
2021-04-29T12:50:24.227Z INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 4.0.6 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-04-29T12:50:24.228Z INFO [CmdLineTool] Loaded plugin: Integrations 4.0.6 [org.graylog.integrations.IntegrationsPlugin]
2021-04-29T12:50:24.228Z INFO [CmdLineTool] Loaded plugin: Collector 4.0.6 [org.graylog.plugins.collector.CollectorPlugin]
2021-04-29T12:50:24.229Z INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 4.0.6 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-04-29T12:50:24.230Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-04-29T12:50:24.230Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-04-29T12:50:24.230Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.6+40b7be5 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-04-29T12:50:24.473Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-04-29T12:50:24.715Z INFO [Version] HV000001: Hibernate Validator null
2021-04-29T12:50:28.408Z INFO [InputBufferImpl] Message journal is enabled.
2021-04-29T12:50:28.428Z INFO [NodeId] Node ID: a032cabe-3276-47a5-8d2b-ca2bd6f0610f
2021-04-29T12:50:28.631Z INFO [LogManager] Loading logs.
2021-04-29T12:50:28.656Z WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index…
2021-04-29T12:50:28.683Z INFO [LogManager] Logs loading complete.
2021-04-29T12:50:28.686Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-04-29T12:50:28.709Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-04-29T12:50:28.759Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-04-29T12:50:28.790Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:316}] to localhost:27017
2021-04-29T12:50:28.794Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 24]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2749544}
2021-04-29T12:50:28.817Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:317}] to localhost:27017
2021-04-29T12:50:29.021Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.

[reimlima (Dummy)]

Hi @kesifalaraskey

Can you filter your log for “ERROR” and paste it here once again…those “infos” and “warns” helps almost nothing to troubleshoot your problem…

[kesifalaraskey (Hafiz Shaikh)]

@reimlima

filter your log for “ERROR”

No ERROR line is showing in logs

[reimlima (Dummy)]

From Graylog Server, can you access (ping, telnet) your ES in AWS?

[gsmith]

@kesifalaraskey

Is this a duplicate post?

Graylog server is not starting

I have installed step to step from documentation but graylog server is not starting. Mongodb server is also running properly graylog/server.log > ubuntu@ip-172-31-42-22:~$ tail -f /var/log/graylog-server/server.log at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_282] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_282] at jav…

[kesifalaraskey (Hafiz Shaikh)]

No, some difference is there between two.
This one has clear view on what action error is occuring and also all other things are working properly just on adding AWS ES endpoint in conf file the thing stop working.

[kesifalaraskey (Hafiz Shaikh)]

How can i do that?
because after adding AWS ES endpoint the server is not starting and also server.log stuck at
this last line

@kesifalaraskey

[InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.

[gsmith]

Oh I see, For a minute I thought it was the same problem. My apologies.

[gsmith]

@kesifalaraskey

Hello,

Can you show your Graylog configuration, make sure you keep private configuration private.

This might help with server.conf file.

grep -v "^#\|^$" server.conf | sed -e "s/#.*$//g"

Also your Elasticsearch configuration file.

grep -v "^#\|^$" elasticsearch.yml | sed -e "s/#.*$//g"

This will help us find your issue or at least narrow it down.

Thanks

Edit:
Make sure ES is running first and in “Green” as I showed in the other post.
Then install/start MongoDb. Last start Graylog Service. Not sure if that will help but doesnt hurt to try.

[kesifalaraskey (Hafiz Shaikh)]

Thanks you all,

I have fixed the thing just by adding elasticsearch version manually in server.conf file.