Graylog is restarting...

Hi,

we have configured graylog server but after few days it is continuously showing that Graylog is restarting…

please provide resolution for the same and also note the status of running services…

run: elasticsearch: (pid 14963) 2s; run: log: (pid 789) 56868s
run: etcd: (pid 810) 56868s; run: log: (pid 794) 56868s
run: graylog-server: (pid 19135) 11958s; run: log: (pid 787) 56868s
down: mongodb: 1s, normally up, want up; run: log: (pid 786) 56868s
run: nginx: (pid 793) 56868s; run: log: (pid 788) 56868s

What’s in the logs of your Graylog node?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#omnibus-package

Hi,

below is the logs…

2018-03-25_21:36:09.29543       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-03-25_21:36:09.29646       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
2018-03-25_21:36:09.29775       at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
2018-03-25_21:36:09.30009       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
2018-03-25_21:36:09.30154       ... 11 more
2018-03-25_21:36:09.30255 WARN  [NodeChecker] Removing host http://192.168.0.72:9200
2018-03-25_21:36:09.30524 INFO  [NodeChecker] Discovered server pool is now:
2018-03-25_21:36:09.81118 INFO  [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-03-25_21:36:39.13958 ERROR [NodeChecker] Connect exception executing NodesInfo!
2018-03-25_21:36:39.14047 io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://192.168.0.72:9200
2018-03-25_21:36:39.14201       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
2018-03-25_21:36:39.14291       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:64) ~[graylog.jar:?]
2018-03-25_21:36:39.14459       at io.searchbox.client.config.discovery.NodeChecker.runOneIteration(NodeChecker.java:65) [graylog.jar:?]
2018-03-25_21:36:39.14691       at com.google.common.util.concurrent.AbstractScheduledService$ServiceDelegate$Task.run(AbstractScheduledService.java:188) [graylog.jar:?]
2018-03-25_21:36:39.14760       at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
2018-03-25_21:36:39.14920       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
2018-03-25_21:36:39.15258       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_151]
2018-03-25_21:36:39.15354       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_151]
2018-03-25_21:36:39.15528       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_151]
2018-03-25_21:36:39.15893       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
2018-03-25_21:36:39.16022       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
2018-03-25_21:36:39.16248       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-03-25_21:36:39.16433 Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.0.72:9200 [/192.168.0.72] failed: Connection refused (Connection refused)
2018-03-25_21:36:39.16628       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
2018-03-25_21:36:39.16710       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) ~[graylog.jar:?]
2018-03-25_21:36:39.16858       at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-03-25_21:36:39.17044       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-03-25_21:36:39.17259       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-03-25_21:36:39.17328       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
2018-03-25_21:36:39.17640       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
2018-03-25_21:36:39.18763       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-03-25_21:36:39.18856       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-03-25_21:36:39.19129       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
2018-03-25_21:36:39.20289       at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
2018-03-25_21:36:39.20442       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
2018-03-25_21:36:39.20563       ... 11 more
2018-03-25_21:36:39.20785 Caused by: java.net.ConnectException: Connection refused (Connection refused)
2018-03-25_21:36:39.21027       at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_151]
2018-03-25_21:36:39.21288       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_151]

it looks like Graylog is not able to connect to elasticsearch.

Did you checked if elasticsearch is running? Did you checked the elasticsearch logfile?

with kind regards
Jan

Yes i checked the elasticsearch log file but could not identify the issue…

pls check the below logs…

2018-03-27_17:47:01.74461 vm.max_map_count = 262144
2018-03-27_17:47:08.48828 [INFO ][o.e.n.Node ] [] initializing …
2018-03-27_17:47:08.49312 2018-03-27 17:47:08,489 main ERROR Unable to write to stream /var/log/graylog/elasticsearch/graylog.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/graylog/elasticsearch/graylog.log
2018-03-27_17:47:08.50850 2018-03-27 17:47:08,492 main ERROR An exception occurred processing Appender rolling org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/graylog/elasticsearch/graylog.log
2018-03-27_17:47:08.51479 at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:264)
2018-03-27_17:47:08.52360 at org.apache.logging.log4j.core.appender.FileManager.writeToDestination(FileManager.java:261)
2018-03-27_17:47:08.53609 at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.writeToDestination(RollingFileManager.java:219)
2018-03-27_17:47:08.53950 at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:294)
2018-03-27_17:47:08.54465 at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:303)
2018-03-27_17:47:08.54984 at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:179)
2018-03-27_17:47:08.55415 at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
2018-03-27_17:47:08.56402 at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
2018-03-27_17:47:08.56602 at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:308)
2018-03-27_17:47:08.56975 at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
2018-03-27_17:47:08.57448 at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
2018-03-27_17:47:08.57717 at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
2018-03-27_17:47:08.57945 at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
2018-03-27_17:47:08.58361 at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
2018-03-27_17:47:08.58697 at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
2018-03-27_17:47:08.58981 at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
2018-03-27_17:47:08.60252 at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
2018-03-27_17:47:08.60763 at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
2018-03-27_17:47:08.60792 at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
2018-03-27_17:47:08.60906 at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
2018-03-27_17:47:08.61228 at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:102)
2018-03-27_17:47:08.61743 at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2116)
2018-03-27_17:47:08.61908 at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2100)
2018-03-27_17:47:08.62158 at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1994)
2018-03-27_17:47:08.62227 at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1966)
2018-03-27_17:47:08.62457 at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1303)
2018-03-27_17:47:08.62525 at org.elasticsearch.node.Node.(Node.java:254)
2018-03-27_17:47:08.62768 at org.elasticsearch.node.Node.(Node.java:245)
2018-03-27_17:47:08.62914 at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233)
2018-03-27_17:47:08.63272 at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
2018-03-27_17:47:08.63547 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
2018-03-27_17:47:08.63863 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
2018-03-27_17:47:08.64103 at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
2018-03-27_17:47:08.64225 at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
2018-03-27_17:47:08.64510 at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
2018-03-27_17:47:08.64715 at org.elasticsearch.cli.Command.main(Command.java:90)
2018-03-27_17:47:08.65115 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.